Skip to content

fix(ci): pull-ci-redhat-developer-rhdh-main-e2e-osd-gcp-operator-nightly - #4419

Merged
openshift-merge-bot[bot] merged 15 commits into
redhat-developer:mainfrom
Fortune-Ndlovu:RHDHBUGS-1136-e2e-OSD-GCP-with-Operator-deployment-not-starting-fix
Jul 16, 2026
Merged

fix(ci): pull-ci-redhat-developer-rhdh-main-e2e-osd-gcp-operator-nightly#4419
openshift-merge-bot[bot] merged 15 commits into
redhat-developer:mainfrom
Fortune-Ndlovu:RHDHBUGS-1136-e2e-OSD-GCP-with-Operator-deployment-not-starting-fix

Conversation

@Fortune-Ndlovu

@Fortune-Ndlovu Fortune-Ndlovu commented Mar 18, 2026

Copy link
Copy Markdown
Member

Description

fix(ci): pull-ci-redhat-developer-rhdh-main-e2e-osd-gcp-operator-nightly

PR acceptance criteria

Please make sure that the following steps are complete:

  • GitHub Actions are completed and successful
  • Unit Tests are updated and passing
  • E2E Tests are updated and passing
  • Documentation is updated if necessary (requirement for new features)
  • Add a screenshot if the change is UX/UI related

How to test changes / Special notes to the reviewer

@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/test e2e-osd-gcp-operator-nightly

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/test e2e-osd-gcp-operator-nightly

@github-actions

Copy link
Copy Markdown
Contributor

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

@github-actions

Copy link
Copy Markdown
Contributor

The container image build workflow finished with status: cancelled.

@github-actions

Copy link
Copy Markdown
Contributor

The container image build workflow finished with status: cancelled.

@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/test e2e-osd-gcp-operator-nightly

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/test e2e-osd-gcp-operator-nightly

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm

@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/test e2e-osd-gcp-operator-nightly

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/test e2e-osd-gcp-operator-nightly

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/test e2e-osd-gcp-operator-nightly

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/test e2e-osd-gcp-operator-nightly

@github-actions

Copy link
Copy Markdown
Contributor

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/test e2e-osd-gcp-operator-nightly

1 similar comment
@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/test e2e-osd-gcp-operator-nightly

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/test e2e-osd-gcp-operator-nightly

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/test e2e-osd-gcp-operator-nightly

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@Fortune-Ndlovu Fortune-Ndlovu changed the title fix(ci): avoid duplicate orchestrator plugins on OSD-GCP (operator e2e) fix(ci): pull-ci-redhat-developer-rhdh-main-e2e-osd-gcp-operator-nightly Mar 20, 2026
@openshift-ci openshift-ci Bot removed the lgtm label Jul 16, 2026
@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/test images

@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/test e2e-osd-gcp-operator-nightly

@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm

@github-actions

Copy link
Copy Markdown
Contributor

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/test e2e-osd-gcp-operator-nightly

@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm

1 similar comment
@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm

@rhdh-qodo-merge

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (2) 📘 Rule violations (0) 📜 Skill insights (0)

Context used
⚠️ Tickets: not configured — ticket URL found in PR but could not be fetched — check ticket provider credentials

Grey Divider


Action required

1. Orchestrator entries duplicated ✓ Resolved 🐞 Bug ≡ Correctness
Description
helm::merge_values deduplicates plugins by the pre-sed .package value, so base orchestrator
entries using :{{…inherit…}} and OSD-GCP diff entries using @sha256:… are both kept.
fix_ghcr_oci_urls_for_osd_gcp then rewrites the base entries to the same @sha256:… form, and
config::create_dynamic_plugins_config outputs both entries, leaving conflicting disabled values
for the same .package.
Code

.ci/pipelines/jobs/ocp-operator.sh[R62-65]

+    -e 's|red-hat-developer-hub-backstage-plugin-orchestrator:{{ "{{" }}inherit{{ "}}" }}|red-hat-developer-hub-backstage-plugin-orchestrator@sha256:062a536d266bcd76d454fc9fdc0157b99e62074f6d3304578a68f515ced83e64|' \
+    -e 's|red-hat-developer-hub-backstage-plugin-orchestrator-backend:{{ "{{" }}inherit{{ "}}" }}|red-hat-developer-hub-backstage-plugin-orchestrator-backend@sha256:365d67fddeeaa2cf0b0266b012eb3e74fb5d5071b848059662353e62e7f9ceab|' \
+    -e 's|red-hat-developer-hub-backstage-plugin-scaffolder-backend-module-orchestrator:{{ "{{" }}inherit{{ "}}" }}|red-hat-developer-hub-backstage-plugin-scaffolder-backend-module-orchestrator@sha256:5b52adc153afcf79fb134262aa01b1e117085bf59c1a1997ff2e3a6f31c7647a|' \
+    -e 's|red-hat-developer-hub-backstage-plugin-orchestrator-form-widgets:{{ "{{" }}inherit{{ "}}" }}|red-hat-developer-hub-backstage-plugin-orchestrator-form-widgets@sha256:b8d9886e39fa1262bfe1046f6ec5825b8e576c4997c744e0f90f722453f35aec|' \
Relevance

⭐⭐⭐ High

Likely real bug: merge de-dupes pre-sed; sed makes packages identical, leaving conflicting
duplicates in generated config.

PR-#4414
PR-#4397

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The merge step explicitly deduplicates by .package before any sed rewrite; because the base
orchestrator packages include the :{{ "{{" }}inherit{{ "}}" }} suffix while the OSD-GCP diff uses
@sha256:…, they are treated as different packages and both survive the merge. After that, the new
sed rules rewrite the base :{{…inherit…}} forms into @sha256:…, making them identical to the
diff entries; the ConfigMap generator then dumps .global.dynamic as-is without re-deduplication,
so duplicates persist into the final dynamic-plugins.yaml.

.ci/pipelines/lib/helm.sh[44-63]
.ci/pipelines/value_files/values_showcase.yaml[301-347]
.ci/pipelines/value_files/diff-values_showcase_OSD-GCP.yaml[18-27]
.ci/pipelines/jobs/ocp-operator.sh[62-65]
.ci/pipelines/lib/config.sh[89-98]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`helm::merge_values` dedups plugins by `.package` **before** `fix_ghcr_oci_urls_for_osd_gcp` rewrites orchestrator packages from `:{{…inherit…}}` to `@sha256:…`. This creates duplicate `.package` entries (same digest) with conflicting `disabled` values in the generated `dynamic-plugins.yaml`.

### Issue Context
- Base values enable orchestrator with `:{{…inherit…}}`.
- OSD-GCP diff values add disabled orchestrator entries using `@sha256:…`.
- After merge, sed rewrites the base entry to `@sha256:…` too.

### Fix Focus Areas
- .ci/pipelines/jobs/ocp-operator.sh[87-92]
- .ci/pipelines/jobs/ocp-operator.sh[107-112]
- .ci/pipelines/lib/helm.sh[52-57]

### Suggested fix
After running `fix_ghcr_oci_urls_for_osd_gcp`, run a yq-based de-duplication pass again on the merged file, e.g.:
- `.global.dynamic.plugins |= (reverse | unique_by(.package) | reverse)`
so the final merged values (and ConfigMap output) contains only one entry per `.package` with the intended precedence (OSD-GCP diff entry should win).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. Operator CM assumption conflict 🐞 Bug ☼ Reliability
Description
config::merge_osd_gcp_operator_dynamic_plugins hard-waits for and patches a
backstage-dynamic-plugins-* ConfigMap, but repo documentation states that when the Backstage CR
sets dynamicPluginsConfigMapName: dynamic-plugins, that ConfigMap “never gets created”, making
OSD-GCP operator jobs fragile and potentially timeout-prone.
Code

.ci/pipelines/lib/config.sh[R246-256]

+  if ! common::poll_until \
+    "oc get cm -n ${namespace} --no-headers 2>/dev/null | grep -q backstage-dynamic-plugins-" \
+    60 5 "operator backstage-dynamic-plugins ConfigMap in ${namespace}"; then
+    return 1
+  fi
+
+  operator_cm=$(oc get cm -n "$namespace" --no-headers 2> /dev/null | awk '/backstage-dynamic-plugins-/{print $1; exit}')
+  if [[ -z "${operator_cm}" ]]; then
+    log::error "OSD-GCP: could not resolve backstage-dynamic-plugins ConfigMap in ${namespace}"
+    return 1
+  fi
Relevance

⭐⭐⭐ High

Hard polling for backstage-dynamic-plugins-* can timeout if operator never creates it under
dynamicPluginsConfigMapName=dynamic-plugins.

PR-#4414
PR-#4186
PR-#4153

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The new OSD-GCP operator path calls config::merge_osd_gcp_operator_dynamic_plugins, which explicitly
polls for a backstage-dynamic-plugins-* ConfigMap and fails if it is absent. However, the
Backstage CR used in this PR still sets dynamicPluginsConfigMapName: dynamic-plugins, and the
repo’s REFACTORING.md explicitly documents that in this configuration the
backstage-dynamic-plugins-* ConfigMap “never gets created”, which would cause the poll to fail and
break the OSD-GCP flow.

.ci/pipelines/lib/config.sh[233-256]
.ci/pipelines/jobs/ocp-operator.sh[55-75]
.ci/pipelines/resources/rhdh-operator/rhdh-start-osd-gcp.yaml[27-36]
.ci/pipelines/REFACTORING.md[247-255]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`config::merge_osd_gcp_operator_dynamic_plugins` assumes an operator-managed `backstage-dynamic-plugins-*` ConfigMap exists and fails if it does not. This conflicts with repo documentation for the `dynamicPluginsConfigMapName: dynamic-plugins` configuration and can break OSD-GCP operator jobs.

### Issue Context
OSD-GCP operator jobs call this function multiple times in `.ci/pipelines/jobs/ocp-operator.sh`, so a single missing ConfigMap causes the whole job path to fail.

### Fix Focus Areas
- `.ci/pipelines/lib/config.sh[246-256]`
- `.ci/pipelines/lib/config.sh[252-321]`
- `.ci/pipelines/jobs/ocp-operator.sh[66-74]`

### Suggested fix
- Change `config::merge_osd_gcp_operator_dynamic_plugins` to:
 - Prefer patching the ConfigMap actually referenced by the Backstage CR (`dynamicPluginsConfigMapName`) when present (here: `dynamic-plugins`).
 - Only patch `backstage-dynamic-plugins-*` if it exists; otherwise log a clear message and either patch `dynamic-plugins` or return success.
 - Update the function’s comments/docs to match the actual behavior.
- Add a clear error message showing what ConfigMaps exist when neither target is found (to avoid silent retries).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

3. Crash filtering drops non-Backstage 🐞 Bug ☼ Reliability
Description
testing::check_backstage_running filters CrashLoopBackOff pods to only those owned by ReplicaSets
for deployment/backstage-${release_name}, which can ignore CrashLooping pods for other critical
workloads (e.g., ${release_name}-postgresql) and delay failure until the full readiness timeout.
Code

.ci/pipelines/lib/testing.sh[R178-197]

+      # During rollouts, ReplicaSets with desired=0 may still show CrashLoop/Terminating pods; only fail on
+      # pods owned by an RS that still has desired replicas (current rollout targets).
+      local significant_crashes="${crash_pods}"
+      local op_deploy="backstage-${release_name}"
+      if oc get deployment "${op_deploy}" -n "${namespace}" &> /dev/null; then
+        local current_rs
+        current_rs=$(oc get rs -n "${namespace}" --no-headers 2> /dev/null | awk -v d="${op_deploy}-" '$1 ~ "^"d && $2 > 0 {print $1}' || true)
+        if [[ -n "${current_rs}" ]]; then
+          significant_crashes=""
+          while IFS= read -r line; do
+            [[ -z "${line}" ]] && continue
+            local pod_name pod_rs
+            pod_name=$(awk '{print $1}' <<< "${line}")
+            pod_rs=$(oc get pod "${pod_name}" -n "${namespace}" -o jsonpath='{.metadata.ownerReferences[?(@.kind=="ReplicaSet")].name}' 2> /dev/null || true)
+            if [[ -n "${pod_rs}" ]] && printf '%s\n' "${current_rs}" | grep -qx "${pod_rs}"; then
+              significant_crashes+="${line}"$'\n'
+            fi
+          done <<< "${crash_pods}"
+        fi
+      fi
Relevance

⭐⭐⭐ High

CrashLoop fail-fast now only considers ReplicaSets for backstage deployment; other workload crashes
may be ignored.

PR-#4414
PR-#4267
PR-#4366

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The crash probe intentionally collects pods beyond Backstage itself (explicitly includes
postgresql in the fallback grep and ${release_name}-postgresql in the label selector), but the
“significant” filter only retains pods whose owner ReplicaSet name matches the active ReplicaSet(s)
for deployment/backstage-${release_name}. Pods from other deployments/ReplicaSets will not match
and will be silently excluded from fail-fast detection when the Backstage deployment exists.

.ci/pipelines/lib/testing.sh[169-197]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The CrashLoopBackOff fail-fast logic can drop real crashes for non-Backstage workloads because it only considers ReplicaSets belonging to `deployment/backstage-${release_name}`.

### Issue Context
The intent is good (ignore stale pods from ReplicaSets with desired=0 during rollouts), but it should apply to the ReplicaSet that owns each crashing pod (or to that pod’s owning controller), not only Backstage’s ReplicaSets.

### Fix Focus Areas
- `.ci/pipelines/lib/testing.sh[178-197]`

### Suggested fix
- For each crashing pod:
 - Determine its owning ReplicaSet (`pod_rs`).
 - Query that ReplicaSet’s desired replicas (e.g., `oc get rs $pod_rs -o jsonpath='{.spec.replicas}'`) and only treat it as “significant” if replicas > 0.
 - If a pod is not owned by a ReplicaSet (Job/StatefulSet), use the owning controller’s desired replicas/ready status (or keep it significant by default).
- This preserves the rollout-stale-pod suppression without hiding crashes from postgres/redis/other dependencies.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


4. Sed errors ignored ✓ Resolved 🐞 Bug ☼ Reliability
Description
initiate_operator_deployments_osd_gcp calls fix_ghcr_oci_urls_for_osd_gcp without checking its
return code, and this script does not enable set -e, so sed failures can silently continue. That
can leave unresolved {{inherit}} tags and/or missing !<plugin-path> suffixes in the generated
ConfigMap, causing harder-to-debug downstream failures.
Code

.ci/pipelines/jobs/ocp-operator.sh[R89-91]

  helm::merge_values "merge" "${DIR}/value_files/${HELM_CHART_VALUE_FILE_NAME}" "${DIR}/value_files/${HELM_CHART_OSD_GCP_DIFF_VALUE_FILE_NAME}" "/tmp/merged-values_showcase_OSD-GCP.yaml"
+  fix_ghcr_oci_urls_for_osd_gcp "/tmp/merged-values_showcase_OSD-GCP.yaml"
  config::create_dynamic_plugins_config "/tmp/merged-values_showcase_OSD-GCP.yaml" "/tmp/configmap-dynamic-plugins.yaml"
Relevance

⭐⭐ Medium

Team sometimes tolerates non-zero exits in CI via || true; unclear if sed failures should be fatal
here.

PR-#4288
PR-#4414

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The new function returns the sed exit code, but both call sites ignore it. Since ocp-operator.sh
does not enable set -e, a non-zero sed exit status (e.g., missing file, permissions) will not
automatically abort and will proceed to ConfigMap generation/deploy.

.ci/pipelines/jobs/ocp-operator.sh[1-1]
.ci/pipelines/jobs/ocp-operator.sh[51-76]
.ci/pipelines/jobs/ocp-operator.sh[87-92]
.ci/pipelines/jobs/ocp-operator.sh[107-112]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`fix_ghcr_oci_urls_for_osd_gcp` can fail but its return code is ignored, and the script doesn’t use `set -e`. This allows the pipeline to continue with an unmodified merged values file.

### Issue Context
`fix_ghcr_oci_urls_for_osd_gcp` is an essential step for OSD-GCP to avoid unresolved `{{inherit}}` and ghcr-related failures; if it fails, continuing usually makes later failures harder to diagnose.

### Fix Focus Areas
- .ci/pipelines/jobs/ocp-operator.sh[89-92]
- .ci/pipelines/jobs/ocp-operator.sh[109-112]

### Suggested fix
Guard both invocations, e.g.:
```bash
fix_ghcr_oci_urls_for_osd_gcp "/tmp/merged-values_showcase_OSD-GCP.yaml" || return 1
```
(and same for the RBAC merged file), optionally logging a clear error before returning.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Informational

5. Wrong function in comment ✓ Resolved 🐞 Bug ⚙ Maintainability
Description
The OSD-GCP diff values files reference replace_inherit_with_explicit_versions(), but the
implementation added in this PR is fix_ghcr_oci_urls_for_osd_gcp. This comment mismatch makes future
maintenance and troubleshooting harder.
Code

.ci/pipelines/value_files/diff-values_showcase_OSD-GCP.yaml[R11-16]

+    # Override includes to empty: the operator's init container image (rhdh-hub-rhel9)
+    # does not contain dynamic-plugins.default.yaml and the catalog index mechanism
+    # is not available, so {{inherit}} cannot resolve. All plugins must use explicit versions.
+    # The {{inherit}} tags in the base file are replaced with explicit versions by
+    # replace_inherit_with_explicit_versions() in ocp-operator.sh after merging.
+    includes: []
Relevance

⭐⭐⭐ High

Comment refers to non-existent helper; team generally accepts small maintainability fixes in CI
scripts/files.

PR-#4414
PR-#4133

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Both OSD-GCP diff values files mention a non-existent helper, while the only helper added in this
change is fix_ghcr_oci_urls_for_osd_gcp in ocp-operator.sh.

.ci/pipelines/value_files/diff-values_showcase_OSD-GCP.yaml[11-16]
.ci/pipelines/value_files/diff-values_showcase-rbac_OSD-GCP.yaml[11-16]
.ci/pipelines/jobs/ocp-operator.sh[46-76]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
Comments in the OSD-GCP diff values files reference `replace_inherit_with_explicit_versions()`, but the code uses `fix_ghcr_oci_urls_for_osd_gcp()`.

### Issue Context
This is documentation-only but can confuse future debugging.

### Fix Focus Areas
- .ci/pipelines/value_files/diff-values_showcase_OSD-GCP.yaml[11-16]
- .ci/pipelines/value_files/diff-values_showcase-rbac_OSD-GCP.yaml[11-16]

### Suggested fix
Change the comment to reference `fix_ghcr_oci_urls_for_osd_gcp()` (or rename the function to match the comment).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Previous review results

Review updated until commit 8df3897

Results up to commit e3d46d2


🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)


Action required
1. Orchestrator entries duplicated ✓ Resolved 🐞 Bug ≡ Correctness
Description
helm::merge_values deduplicates plugins by the pre-sed .package value, so base orchestrator
entries using :{{…inherit…}} and OSD-GCP diff entries using @sha256:… are both kept.
fix_ghcr_oci_urls_for_osd_gcp then rewrites the base entries to the same @sha256:… form, and
config::create_dynamic_plugins_config outputs both entries, leaving conflicting disabled values
for the same .package.
Code

.ci/pipelines/jobs/ocp-operator.sh[R62-65]

+    -e 's|red-hat-developer-hub-backstage-plugin-orchestrator:{{ "{{" }}inherit{{ "}}" }}|red-hat-developer-hub-backstage-plugin-orchestrator@sha256:062a536d266bcd76d454fc9fdc0157b99e62074f6d3304578a68f515ced83e64|' \
+    -e 's|red-hat-developer-hub-backstage-plugin-orchestrator-backend:{{ "{{" }}inherit{{ "}}" }}|red-hat-developer-hub-backstage-plugin-orchestrator-backend@sha256:365d67fddeeaa2cf0b0266b012eb3e74fb5d5071b848059662353e62e7f9ceab|' \
+    -e 's|red-hat-developer-hub-backstage-plugin-scaffolder-backend-module-orchestrator:{{ "{{" }}inherit{{ "}}" }}|red-hat-developer-hub-backstage-plugin-scaffolder-backend-module-orchestrator@sha256:5b52adc153afcf79fb134262aa01b1e117085bf59c1a1997ff2e3a6f31c7647a|' \
+    -e 's|red-hat-developer-hub-backstage-plugin-orchestrator-form-widgets:{{ "{{" }}inherit{{ "}}" }}|red-hat-developer-hub-backstage-plugin-orchestrator-form-widgets@sha256:b8d9886e39fa1262bfe1046f6ec5825b8e576c4997c744e0f90f722453f35aec|' \
Relevance

⭐⭐⭐ High

Likely real bug: merge de-dupes pre-sed; sed makes packages identical, leaving conflicting
duplicates in generated config.

PR-#4414
PR-#4397

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The merge step explicitly deduplicates by .package before any sed rewrite; because the base
orchestrator packages include the :{{ "{{" }}inherit{{ "}}" }} suffix while the OSD-GCP diff uses
@sha256:…, they are treated as different packages and both survive the merge. After that, the new
sed rules rewrite the base :{{…inherit…}} forms into @sha256:…, making them identical to the
diff entries; the ConfigMap generator then dumps .global.dynamic as-is without re-deduplication,
so duplicates persist into the final dynamic-plugins.yaml.

.ci/pipelines/lib/helm.sh[44-63]
.ci/pipelines/value_files/values_showcase.yaml[301-347]
.ci/pipelines/value_files/diff-values_showcase_OSD-GCP.yaml[18-27]
.ci/pipelines/jobs/ocp-operator.sh[62-65]
.ci/pipelines/lib/config.sh[89-98]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`helm::merge_values` dedups plugins by `.package` **before** `fix_ghcr_oci_urls_for_osd_gcp` rewrites orchestrator packages from `:{{…inherit…}}` to `@sha256:…`. This creates duplicate `.package` entries (same digest) with conflicting `disabled` values in the generated `dynamic-plugins.yaml`.

### Issue Context
- Base values enable orchestrator with `:{{…inherit…}}`.
- OSD-GCP diff values add disabled orchestrator entries using `@sha256:…`.
- After merge, sed rewrites the base entry to `@sha256:…` too.

### Fix Focus Areas
- .ci/pipelines/jobs/ocp-operator.sh[87-92]
- .ci/pipelines/jobs/ocp-operator.sh[107-112]
- .ci/pipelines/lib/helm.sh[52-57]

### Suggested fix
After running `fix_ghcr_oci_urls_for_osd_gcp`, run a yq-based de-duplication pass again on the merged file, e.g.:
- `.global.dynamic.plugins |= (reverse | unique_by(.package) | reverse)`
so the final merged values (and ConfigMap output) contains only one entry per `.package` with the intended precedence (OSD-GCP diff entry should win).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended
2. Sed errors ignored ✓ Resolved 🐞 Bug ☼ Reliability
Description
initiate_operator_deployments_osd_gcp calls fix_ghcr_oci_urls_for_osd_gcp without checking its
return code, and this script does not enable set -e, so sed failures can silently continue. That
can leave unresolved {{inherit}} tags and/or missing !<plugin-path> suffixes in the generated
ConfigMap, causing harder-to-debug downstream failures.
Code

.ci/pipelines/jobs/ocp-operator.sh[R89-91]

  helm::merge_values "merge" "${DIR}/value_files/${HELM_CHART_VALUE_FILE_NAME}" "${DIR}/value_files/${HELM_CHART_OSD_GCP_DIFF_VALUE_FILE_NAME}" "/tmp/merged-values_showcase_OSD-GCP.yaml"
+  fix_ghcr_oci_urls_for_osd_gcp "/tmp/merged-values_showcase_OSD-GCP.yaml"
  config::create_dynamic_plugins_config "/tmp/merged-values_showcase_OSD-GCP.yaml" "/tmp/configmap-dynamic-plugins.yaml"
Relevance

⭐⭐ Medium

Team sometimes tolerates non-zero exits in CI via || true; unclear if sed failures should be fatal
here.

PR-#4288
PR-#4414

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The new function returns the sed exit code, but both call sites ignore it. Since ocp-operator.sh
does not enable set -e, a non-zero sed exit status (e.g., missing file, permissions) will not
automatically abort and will proceed to ConfigMap generation/deploy.

.ci/pipelines/jobs/ocp-operator.sh[1-1]
.ci/pipelines/jobs/ocp-operator.sh[51-76]
.ci/pipelines/jobs/ocp-operator.sh[87-92]
.ci/pipelines/jobs/ocp-operator.sh[107-112]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`fix_ghcr_oci_urls_for_osd_gcp` can fail but its return code is ignored, and the script doesn’t use `set -e`. This allows the pipeline to continue with an unmodified merged values file.

### Issue Context
`fix_ghcr_oci_urls_for_osd_gcp` is an essential step for OSD-GCP to avoid unresolved `{{inherit}}` and ghcr-related failures; if it fails, continuing usually makes later failures harder to diagnose.

### Fix Focus Areas
- .ci/pipelines/jobs/ocp-operator.sh[89-92]
- .ci/pipelines/jobs/ocp-operator.sh[109-112]

### Suggested fix
Guard both invocations, e.g.:
```bash
fix_ghcr_oci_urls_for_osd_gcp "/tmp/merged-values_showcase_OSD-GCP.yaml" || return 1
```
(and same for the RBAC merged file), optionally logging a clear error before returning.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Informational
3. Wrong function in comment ✓ Resolved 🐞 Bug ⚙ Maintainability
Description
The OSD-GCP diff values files reference replace_inherit_with_explicit_versions(), but the
implementation added in this PR is fix_ghcr_oci_urls_for_osd_gcp. This comment mismatch makes future
maintenance and troubleshooting harder.
Code

.ci/pipelines/value_files/diff-values_showcase_OSD-GCP.yaml[R11-16]

+    # Override includes to empty: the operator's init container image (rhdh-hub-rhel9)
+    # does not contain dynamic-plugins.default.yaml and the catalog index mechanism
+    # is not available, so {{inherit}} cannot resolve. All plugins must use explicit versions.
+    # The {{inherit}} tags in the base file are replaced with explicit versions by
+    # replace_inherit_with_explicit_versions() in ocp-operator.sh after merging.
+    includes: []
Relevance

⭐⭐⭐ High

Comment refers to non-existent helper; team generally accepts small maintainability fixes in CI
scripts/files.

PR-#4414
PR-#4133

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Both OSD-GCP diff values files mention a non-existent helper, while the only helper added in this
change is fix_ghcr_oci_urls_for_osd_gcp in ocp-operator.sh.

.ci/pipelines/value_files/diff-values_showcase_OSD-GCP.yaml[11-16]
.ci/pipelines/value_files/diff-values_showcase-rbac_OSD-GCP.yaml[11-16]
.ci/pipelines/jobs/ocp-operator.sh[46-76]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
Comments in the OSD-GCP diff values files reference `replace_inherit_with_explicit_versions()`, but the code uses `fix_ghcr_oci_urls_for_osd_gcp()`.

### Issue Context
This is documentation-only but can confuse future debugging.

### Fix Focus Areas
- .ci/pipelines/value_files/diff-values_showcase_OSD-GCP.yaml[11-16]
- .ci/pipelines/value_files/diff-values_showcase-rbac_OSD-GCP.yaml[11-16]

### Suggested fix
Change the comment to reference `fix_ghcr_oci_urls_for_osd_gcp()` (or rename the function to match the comment).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Qodo Logo

@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm keeps failing: dial tcp: lookup api.rhdh-4-18-XXXXXXXXX-sxs4k.XXXXXXX.devcluster.openshift.com on 172.30.0.10:53: no such host
dial tcp: lookup api.rhdh-4-18-XXXXXXXXX-sxs4k.XXXXXXX.devcluster.openshift.com on 172.30.0.10:53: no such host

@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm

@github-actions

Copy link
Copy Markdown
Contributor

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

@sonarqubecloud

Copy link
Copy Markdown

@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm

@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/test e2e-osd-gcp-operator-nightly

@subhashkhileri subhashkhileri left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@openshift-ci openshift-ci Bot added the lgtm label Jul 16, 2026
@openshift-ci

openshift-ci Bot commented Jul 16, 2026

Copy link
Copy Markdown

@Fortune-Ndlovu: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-osd-gcp-operator-nightly 6528c43 link false /test e2e-osd-gcp-operator-nightly

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@Fortune-Ndlovu

Copy link
Copy Markdown
Member Author

/override e2e-osd-gcp-operator-nightly

@openshift-ci

openshift-ci Bot commented Jul 16, 2026

Copy link
Copy Markdown

@Fortune-Ndlovu: Fortune-Ndlovu unauthorized: /override is restricted to Repo administrators, approvers in top level OWNERS file, and the following github teams:.

Details

In response to this:

/override e2e-osd-gcp-operator-nightly

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@openshift-merge-bot
openshift-merge-bot Bot merged commit e7dc80c into redhat-developer:main Jul 16, 2026
15 of 16 checks passed
@Fortune-Ndlovu
Fortune-Ndlovu deleted the RHDHBUGS-1136-e2e-OSD-GCP-with-Operator-deployment-not-starting-fix branch July 16, 2026 15:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants