Skip to content

refactor(e2e): refactor runtime tests into TypeScript - #4809

Merged
openshift-merge-bot[bot] merged 22 commits into
redhat-developer:mainfrom
zdrapela:fix/showcase-runtime-test-ordering
Jun 26, 2026
Merged

refactor(e2e): refactor runtime tests into TypeScript#4809
openshift-merge-bot[bot] merged 22 commits into
redhat-developer:mainfrom
zdrapela:fix/showcase-runtime-test-ordering

Conversation

@zdrapela

@zdrapela zdrapela commented May 14, 2026

Copy link
Copy Markdown
Member

Goal

Consolidate SHOWCASE_RUNTIME_DB into SHOWCASE_RUNTIME — a single Playwright project that deploys RHDH from TypeScript (runtime-deploy.ts) and runs all runtime tests sequentially (workers: 1).

What the runtime tests actually test

All 4 test files are infrastructure-level tests that only need a basic RHDH instance up and serving the UI:

Test What it validates
config-map.spec.ts ConfigMap app.title change propagates after deployment restart
verify-schema-mode.spec.ts RHDH boots with pluginDivisionMode: schema on a restricted (NOCREATEDB) DB user
verify-tls-config-with-external-rds.spec.ts RHDH connects to 4 AWS RDS PostgreSQL versions over TLS
verify-tls-config-with-external-azure-db.spec.ts RHDH connects to 4 Azure DB PostgreSQL versions over TLS

None exercise catalog, scaffolder, TechDocs, search, Kubernetes, ArgoCD, or any other domain-specific plugin. They only need guest auth and the core backend serving the UI.

Key changes

Single source of truth (runtime-config.ts)

All deployment configuration is generated from a single TypeScript module — no static YAML files. The same config produces both Helm values and Operator Backstage CR, ensuring the two install methods stay in sync.

  • Helm path: generates values YAML as a temp file with only chart-default overrides + --set flags for image/cluster. Lightspeed disabled via global.lightspeed.enabled: false. PVC for dynamic-plugins-root persists plugins across restarts.
  • Operator path: generates app-config ConfigMap, dynamic-plugins ConfigMap (includes: [], plugins: []), and Backstage CR programmatically. Lightspeed disabled via flavours: [].
  • CATALOG_INDEX_IMAGE: opt-in override. Helm uses global.catalogIndex.image.* (matching CI's helm::get_image_params()); Operator pushes env var with containers: ["install-dynamic-plugins"]. When not set, chart/operator defaults apply.

Deployment moved from shell to TypeScript

CI scripts (ocp-nightly.sh, ocp-operator.sh) are reduced to thin wrappers that set env vars and invoke Playwright. All deployment logic lives in runtime-deploy.ts:

  • ensureRuntimeDeployed() — idempotent, called from the first test's beforeAll
  • Supports both Helm (helm upgrade -i) and Operator (Backstage CR) install methods
  • Discovers PostgreSQL service/secret for schema-mode env vars
  • External DB tests (postgres-config.ts) reconfigure the running instance at runtime

Shared utilities extracted

  • resolveInstallMethod(), base64Encode/Decode()helper.ts
  • patchAppConfig(), restartDeploymentWithRetry(), jsonPatchDeployment()KubeClient
  • AppConfigYaml interface, ImageRef type, imageRefToString()runtime-config.ts
  • getDeploymentName() in schema-mode-setup now delegates to getRhdhDeploymentName()
  • run(), discoverRouterBase() exported from runtime-deploy.ts

Files deleted (replaced by TypeScript generation)

  • .ci/pipelines/resources/postgres-db/values-showcase-postgres.yaml → generated by generateHelmValuesYaml()
  • .ci/pipelines/resources/postgres-db/rds-app-config.yaml → generated by generateAppConfigYaml()
  • .ci/pipelines/resources/rhdh-operator/rhdh-start-runtime.yaml → generated by generateBackstageCR()

Jira: https://redhat.atlassian.net/browse/RHIDP-9140
Jira: https://redhat.atlassian.net/browse/RHIDP-9141

@openshift-ci

openshift-ci Bot commented May 14, 2026

Copy link
Copy Markdown

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@zdrapela

Copy link
Copy Markdown
Member Author

/review

@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-operator-nightly

@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-v4-18-helm-nightly

@rhdh-qodo-merge

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Warning

/review is deprecated. Use /agentic_review instead (removal date not yet scheduled).

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 PR contains tests
🔒 Security concerns

TLS verification disabled:
.ci/pipelines/resources/rhdh-operator/rhdh-start-runtime-local.yaml sets NODE_TLS_REJECT_UNAUTHORIZED=0, which disables TLS certificate verification for Node.js. This weakens transport security and can hide real certificate/hostname problems. It should be avoided or tightly scoped to CI-only scenarios, preferring proper CA configuration (e.g., postgres-crt + NODE_EXTRA_CA_CERTS) instead.

⚡ Recommended focus areas for review

Security

The operator CR injects NODE_TLS_REJECT_UNAUTHORIZED=0, which disables TLS certificate verification for Node.js. This can mask real TLS issues in the external DB tests and is risky if it ever leaks beyond CI/test contexts. Consider removing it, scoping it to only the specific test pod/job that needs it, or relying on NODE_EXTRA_CA_CERTS/proper CA injection instead.

    envs:
      - name: NODE_TLS_REJECT_UNAUTHORIZED
        value: "0"
    secrets:
      - name: rhdh-runtime-config
  route:
    enabled: true
# Disable all default flavours (e.g. lightspeed) to avoid unnecessary sidecar
# containers and init containers that slow down startup and restarts.
# Runtime tests don't need lightspeed — they only test ConfigMap changes and DB connectivity.
flavours: []
📚 Focus areas based on broader codebase context

Possible Issue

prepareForExternalDatabase() patches app-config to set backend.database with only a connection block (host/port/user/password) and omits key fields like client, database (name), and any SSL configuration. This may leave runtime behavior dependent on whatever defaults/previous config exists, and can cause external DB tests to fail or be flaky depending on the base deployment config. (Ref 4)

export async function prepareForExternalDatabase(
  kubeClient: KubeClient,
  namespace: string,
  deploymentName: string,
): Promise<void> {
  // --- 1. Remove stale POSTGRES_* env vars patched onto the deployment ---
  // Schema-mode tests may have added individual secretKeyRef env vars pointing
  // to a *-postgresql secret. These override the bulk envFrom injection from
  // postgres-cred and must be removed before external DB tests.
  await removeSchemaModePatchedEnvVars(kubeClient, deploymentName, namespace);

  // --- 2. Patch app-config ConfigMap to use external DB connection ---
  const configMapName = await kubeClient.findAppConfigMap(namespace);
  const configMapResponse = await kubeClient.getConfigMap(
    configMapName,
    namespace,
  );
  const configMap = configMapResponse.body;
  const configKey = Object.keys(configMap.data || {}).find((key) =>
    key.includes("app-config"),
  );

  if (!configKey || !configMap.data) {
    throw new Error(
      `No app-config data key found in ConfigMap '${configMapName}'`,
    );
  }

  const appConfig = yaml.load(configMap.data[configKey]) as AppConfigYaml;

  console.log(
    "Patching app-config to use external database connection (env var placeholders)...",
  );
  appConfig.backend = appConfig.backend || {};
  appConfig.backend.database = {
    connection: {
      host: "${POSTGRES_HOST}",
      port: "${POSTGRES_PORT}",
      user: "${POSTGRES_USER}",
      password: "${POSTGRES_PASSWORD}",
    },
  };

  configMap.data[configKey] = yaml.dump(appConfig);
  delete configMap.metadata?.creationTimestamp;
  delete configMap.metadata?.resourceVersion;

  await kubeClient.coreV1Api.replaceNamespacedConfigMap(
    configMapName,
    namespace,
    configMap,
  );
  console.log("App-config patched for external database connection");

  // --- 3. Add POSTGRES_* env vars to the deployment via secretKeyRef ---
  // The deployment starts with internal DB (no postgres-cred env vars).
  // Add individual env vars pointing to the postgres-cred secret so the
  // app-config ${POSTGRES_HOST} etc. placeholders resolve correctly.
  await ensurePostgresCredEnvVars(kubeClient, deploymentName, namespace);
}

Reference reasoning: The existing schema-mode setup shows a more complete and explicit backend.database config, including client: "pg", connection.database: "${POSTGRES_DB}", and ssl settings. Aligning the external-db patch to include these fields would make the external DB switch deterministic and consistent with how the repo configures DB connections elsewhere.

📄 References
  1. redhat-developer/rhdh/e2e-tests/playwright/e2e/external-database/verify-tls-config-with-external-crunchy.spec.ts [1-37]
  2. redhat-developer/rhdh/e2e-tests/playwright.config.ts [205-217]
  3. redhat-developer/rhdh/e2e-tests/playwright/e2e/plugin-division-mode-schema/verify-schema-mode.spec.ts [78-186]
  4. redhat-developer/rhdh/e2e-tests/playwright/e2e/plugin-division-mode-schema/schema-mode-setup.ts [27-354]

@zdrapela

Copy link
Copy Markdown
Member Author

/agentic_review

@rhdh-qodo-merge

rhdh-qodo-merge Bot commented May 14, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (4) 📘 Rule violations (0) 📎 Requirement gaps (0) 📜 Skill insights (0)

Context used
✅ Tickets: RHIDP-9140

Grey Divider


Action required

1. KubeClient ignores env token 🐞 Bug ☼ Reliability
Description
KubeClient now calls loadFromDefault() and no longer builds a client config from
K8S_CLUSTER_URL/K8S_CLUSTER_TOKEN, so Playwright utilities may authenticate using the developer’s
current kubeconfig identity instead of the exported service-account token. This breaks
local-test-setup flows (and any environment relying on those env vars) with RBAC/permission failures
during namespace/deployment/configmap mutations.
Code

e2e-tests/playwright/utils/kube-client.ts[R123-129]

  constructor() {
    try {
      this.kc = new k8s.KubeConfig();
-      this.kc.loadFromOptions({
-        clusters: [
-          {
-            name: "my-openshift-cluster",
-            server: process.env.K8S_CLUSTER_URL,
-            skipTLSVerify: true,
-          },
-        ],
-        users: [
-          {
-            name: "ci-user",
-            token: process.env.K8S_CLUSTER_TOKEN,
-          },
-        ],
-        contexts: [
-          {
-            name: "default-context",
-            user: "ci-user",
-            cluster: "my-openshift-cluster",
-          },
-        ],
-        currentContext: "default-context",
-      });
+      this.kc.loadFromDefault();

      this.appsApi = this.kc.makeApiClient(k8s.AppsV1Api);
      this.coreV1Api = this.kc.makeApiClient(k8s.CoreV1Api);
Relevance

⭐⭐⭐ High

Recent work depends on exported K8S_CLUSTER_TOKEN for local/CI; likely to restore env-token auth. PR
4542.

PR-#4542
PR-#4023

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The local headed setup explicitly exports a fresh service-account token for tests, but does not
update kubeconfig with that token. With loadFromDefault(), KubeClient no longer uses the
exported token and will instead use the user’s existing kubeconfig identity, which can lack the
required permissions for runtime deployment and config patching.

e2e-tests/playwright/utils/kube-client.ts[123-131]
e2e-tests/local-test-setup.sh[98-121]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`KubeClient` was changed to always use `kc.loadFromDefault()`, which ignores `K8S_CLUSTER_URL`/`K8S_CLUSTER_TOKEN`. The local headed workflow (`e2e-tests/local-test-setup.sh`) exports a fresh service-account token but does **not** log in / rewrite kubeconfig, so the TS code now runs as whatever user/context is already in kubeconfig and can fail on cluster-admin operations.

## Issue Context
The repo already provisions and exports `K8S_CLUSTER_URL`/`K8S_CLUSTER_TOKEN` specifically for test automation. `KubeClient` should keep supporting those env vars as the primary auth mechanism when present, and only fall back to default kubeconfig when they’re absent.

## Fix Focus Areas
- e2e-tests/playwright/utils/kube-client.ts[123-137]

### Concrete fix
- In `KubeClient` constructor:
 - If both `process.env.K8S_CLUSTER_URL` and `process.env.K8S_CLUSTER_TOKEN` are set, call `kc.loadFromOptions(...)` (the previous behavior) using those values.
 - Else, call `kc.loadFromDefault()`.
- Keep `skipTLSVerify: true` parity with the previous options-based config (or make it configurable if needed).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

2. Duplicate plugin disable entries 🐞 Bug ≡ Correctness
Description
run_operator_runtime_config_change_tests() appends new dynamic-plugin entries with disabled: true
for GitHub/Keycloak modules, but those same packages already exist in the generated config (from
values_showcase.yaml) with disabled: false, creating conflicting duplicates. Depending on how
the dynamic plugin loader resolves duplicates, the plugins may remain enabled and still fail at
runtime due to missing configuration in the minimal runtime namespace.
Code

.ci/pipelines/jobs/ocp-operator.sh[R104-111]

+  # Disable plugins that require external configuration (GitHub org, Keycloak, etc.)
+  # not available in the minimal runtime namespace.
+  cat >> /tmp/configmap-dynamic-plugins-runtime.yaml << 'RUNTIME_PLUGINS_EOF'
+      - package: ./dynamic-plugins/dist/backstage-plugin-catalog-backend-module-github-dynamic
+        disabled: true
+      - package: ./dynamic-plugins/dist/backstage-community-plugin-catalog-backend-module-keycloak-dynamic
+        disabled: true
+RUNTIME_PLUGINS_EOF
Relevance

⭐⭐⭐ High

Team previously fixed CI issues from duplicate/Conflicting dynamic-plugin entries by adding
dedupe/override logic in values merging.

PR-#2417
PR-#4791

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The runtime dynamic-plugins ConfigMap is generated from values_showcase.yaml (per
HELM_CHART_VALUE_FILE_NAME) and that file already includes both packages with disabled: false;
the operator job then appends the same packages with disabled: true, resulting in duplicates with
conflicting flags.

.ci/pipelines/jobs/ocp-operator.sh[85-114]
.ci/pipelines/env_variables.sh[28-32]
.ci/pipelines/lib/config.sh[73-99]
.ci/pipelines/value_files/values_showcase.yaml[20-49]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`ocp-operator.sh` generates the runtime `dynamic-plugins` ConfigMap from `values_showcase.yaml` (which already includes the GitHub + Keycloak catalog backend modules as `disabled: false`) and then appends *additional* entries for the same packages marked `disabled: true`. This produces conflicting duplicate entries for the same package and relies on unspecified “last one wins” behavior.

## Issue Context
- Runtime dynamic-plugins ConfigMap is produced from `.global.dynamic` in the base values file, so the runtime file already contains these plugin entries.
- Appending duplicates makes the effective state ambiguous and can result in the plugins still being enabled.

## Fix Focus Areas
- .ci/pipelines/jobs/ocp-operator.sh[103-112]
- .ci/pipelines/value_files/values_showcase.yaml[20-49]
- .ci/pipelines/env_variables.sh[28-31]
- .ci/pipelines/lib/config.sh[73-99]

## Suggested fix approach
1. After generating `/tmp/configmap-dynamic-plugins-runtime.yaml`, *edit* the YAML to set `disabled: true` for existing entries whose `.package` matches the two packages (use `yq`), rather than appending new list items.
2. Alternatively, generate runtime dynamic-plugins from a dedicated runtime values file (with those packages removed/disabled) instead of mutating the generated ConfigMap via `cat >>`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. PVC failures treated transient 🐞 Bug ☼ Reliability
Description
checkPodFailureStates() treats PodScheduled=False as transient when the condition message contains
the substring persistentvolumeclaim, which can match genuine unbound-PVC scheduling failures and
suppress fast-fail detection. This can convert real infra failures into long waits until the outer
deployment-ready timeout expires.
Code

e2e-tests/playwright/utils/kube-client.ts[R630-640]

+            // Ephemeral volume PVC provisioning causes a brief Unschedulable state
+            // that resolves once the PVC is created. Treat as transient, not fatal.
+            if (
+              condition.message?.includes("ephemeral volume controller") ||
+              condition.message?.includes("persistentvolumeclaim")
+            ) {
+              console.log(
+                `Pod ${podName} waiting for ephemeral volume provisioning (transient)`,
+              );
+              continue;
+            }
Relevance

⭐⭐ Medium

Repo favors fast-fail pod scheduling/startup errors, but no prior guidance on treating
'persistentvolumeclaim' as transient.

PR-#3830
PR-#4414

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The new conditional explicitly continues (suppresses the error return) for any
PodScheduled=False with a message containing persistentvolumeclaim; since
waitForDeploymentReady() loops until timeout, this can delay failure reporting significantly.

e2e-tests/playwright/utils/kube-client.ts[623-642]
e2e-tests/playwright/utils/kube-client.ts[719-822]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`checkPodFailureStates()` now considers any `PodScheduled=False` condition whose message contains `persistentvolumeclaim` as transient and continues waiting. This substring is broad and can match real scheduling failures (e.g., PVC never binds / no storage class), which should remain fast-fail.

## Issue Context
This logic runs inside `waitForDeploymentReady()`. When a real PVC bind failure is masked, the loop continues until the overall timeout, slowing CI feedback and making failures less actionable.

## Fix Focus Areas
- e2e-tests/playwright/utils/kube-client.ts[623-642]
- e2e-tests/playwright/utils/kube-client.ts[719-822]

## Suggested fix approach
- Restrict the transient condition to a more specific message pattern tied to ephemeral volume provisioning (e.g., match only `ephemeral volume controller`), OR
- Gate the transient path behind additional checks (e.g., only for pods known to use `ephemeral.volumeClaimTemplate`, or only for messages that explicitly indicate PVC creation is in progress), and keep other `persistentvolumeclaim` scheduling failures as fatal.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Informational

4. Runtime deploy reuses stale 🐞 Bug ☼ Reliability
Description
ensureRuntimeDeployed() skips deployment when it finds an existing Deployment with readyReplicas >=
1, but the CI wrappers no longer delete/recreate the runtime namespace before running the project.
This can cause reruns to execute against a previous job’s already-running image/config and return
misleading test results (false positives) or flaky behavior from leftover state.
Code

e2e-tests/playwright/utils/runtime-deploy.ts[R410-462]

+export async function ensureRuntimeDeployed(): Promise<void> {
+  if (deployed) {
+    console.log("Runtime deployment already completed in this process");
+    return;
+  }
+
+  const installMethod = resolveInstallMethod();
+  const routerBase =
+    process.env.K8S_CLUSTER_ROUTER_BASE || (await discoverRouterBase());
+
+  const config = resolveConfig(routerBase);
+  const { namespace, releaseName } = config;
+
+  console.log(
+    `\n=== Runtime deployment (${installMethod}) ===\n` +
+      `  namespace:    ${namespace}\n` +
+      `  releaseName:  ${releaseName}\n` +
+      `  routerBase:   ${routerBase}\n` +
+      `  image:        ${config.image.registry}/${config.image.repository}:${config.image.tag}\n` +
+      (config.catalogIndex
+        ? `  catalogIndex: ${imageRefToString(config.catalogIndex)}\n`
+        : `  catalogIndex: (chart/operator default)\n`),
+  );
+
+  const kubeClient = new KubeClient();
+
+  // Check if deployment already exists and is ready
+  const deploymentName = getRhdhDeploymentName();
+  try {
+    const dep = await kubeClient.appsApi.readNamespacedDeployment(
+      deploymentName,
+      namespace,
+    );
+    const ready = dep.body.status?.readyReplicas ?? 0;
+    if (ready >= 1) {
+      console.log(
+        `Deployment ${deploymentName} already running (${ready} ready replicas) — skipping deploy`,
+      );
+      deployed = true;
+      // Still configure schema-mode env if not already set
+      if (!process.env.SCHEMA_MODE_DB_ADMIN_PASSWORD) {
+        await configureSchemaMode(
+          kubeClient,
+          namespace,
+          releaseName,
+          installMethod,
+        );
+      }
+      return;
+    }
+  } catch {
+    // Deployment doesn't exist — proceed with fresh deploy
+  }
Relevance

⭐ Low

Team often rejects extra teardown/cleanup; idempotent reuse preferred. Similar cleanup request
rejected in PR 3987.

PR-#3987

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The TS deploy helper explicitly returns early when it sees a ready Deployment, and the nightly
runtime wrapper no longer performs any namespace cleanup before invoking Playwright. Together, that
makes runtime tests dependent on whatever is already running in the shared runtime namespace.

e2e-tests/playwright/utils/runtime-deploy.ts[410-462]
.ci/pipelines/jobs/ocp-nightly.sh[49-62]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`ensureRuntimeDeployed()` treats an existing ready deployment as a no-op and returns early. After this PR, `.ci/pipelines/jobs/ocp-nightly.sh` and `.ci/pipelines/jobs/ocp-operator.sh` no longer clean the runtime namespace up-front, so reruns can silently reuse an old deployment (wrong image/tag, old ConfigMaps/Secrets, old DB state).

## Issue Context
Previously the wrapper scripts performed `namespace::configure` + uninstall/reinstall, ensuring a clean environment per CI run. That behavior is now effectively conditional on whether a prior deployment still exists and is ready.

## Fix Focus Areas
- e2e-tests/playwright/utils/runtime-deploy.ts[436-468]
- .ci/pipelines/jobs/ocp-nightly.sh[49-62]

### Concrete fix options
Pick one (preferred is A):

A) Default to clean deploy unless explicitly opted out:
- Add env var like `RUNTIME_DEPLOY_SKIP_IF_READY=true`.
- In CI, do not set it (so runtime deploy always deletes/recreates namespace).
- For local iteration, developers may set it to speed up runs.

B) Validate “same deployment” before skipping:
- If skipping, first verify deployed image tag matches `TAG_NAME` (and install method), otherwise redeploy.

Also consider: even when skipping deploy, still ensure required runtime test prerequisites exist (e.g., placeholder secrets) and set `BASE_URL` if missing.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Qodo Logo

@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm-nightly

@zdrapela
zdrapela force-pushed the fix/showcase-runtime-test-ordering branch from 4ec83ad to d383484 Compare May 14, 2026 13:49
@github-actions

Copy link
Copy Markdown
Contributor

The container image build workflow finished with status: cancelled.

@codecov

codecov Bot commented May 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 54.77%. Comparing base (8ba9f0e) to head (ac701b4).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4809      +/-   ##
==========================================
- Coverage   55.39%   54.77%   -0.62%     
==========================================
  Files         122      110      -12     
  Lines        2365     2147     -218     
  Branches      563      518      -45     
==========================================
- Hits         1310     1176     -134     
+ Misses       1048      970      -78     
+ Partials        7        1       -6     
Flag Coverage Δ
rhdh 54.77% <ø> (-0.62%) ⬇️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8ba9f0e...ac701b4. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@zdrapela
zdrapela force-pushed the fix/showcase-runtime-test-ordering branch from d383484 to af1c911 Compare May 14, 2026 14:06
@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-operator-nightly

@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm-nightly

@github-actions

Copy link
Copy Markdown
Contributor

The container image build workflow finished with status: cancelled.

@zdrapela zdrapela changed the title fix(ci): deploy showcase-runtime with internal DB and fix test ordering fix(ci): deploy showcase-runtime with internal DB and fix test ordering May 14, 2026
@zdrapela zdrapela changed the title fix(ci): deploy showcase-runtime with internal DB and fix test ordering fix(ci): deploy showcase-runtime with internal DB and fix test orderind May 14, 2026
@zdrapela zdrapela changed the title fix(ci): deploy showcase-runtime with internal DB and fix test orderind fix(ci): deploy showcase-runtime with internal DB and fix test ordering May 14, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@zdrapela

Copy link
Copy Markdown
Member Author

/retest

@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-operator-nightly

@zdrapela
zdrapela force-pushed the fix/showcase-runtime-test-ordering branch from af1c911 to 91c4806 Compare May 15, 2026 06:10
@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm-nightly

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@zdrapela
zdrapela force-pushed the fix/showcase-runtime-test-ordering branch from 91c4806 to ce2f25e Compare May 15, 2026 15:34
@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-helm-nightly

zdrapela added 17 commits June 26, 2026 09:59
- Replace 8 inline Buffer.from() base64 calls with base64Encode/Decode
  from helper.ts (keycloak.ts, api-helper.ts, annotator.spec.ts,
  scaffolder-relation-processor.spec.ts)
- Fix method name typo: createCongifmap -> createConfigMap in KubeClient
- Use kubeClient.createConfigMap() in runtime-deploy.ts instead of
  inline coreV1Api.createNamespacedConfigMap() calls

Assisted-by: OpenCode
Move reusable utilities out of runtime-deploy.ts and
runtime-config.ts into shared modules:

helper.ts:
- run() — shell command execution with stdout/stderr capture
- discoverRouterBase() — OpenShift cluster router base discovery
- ImageRef interface, imageRefToString(), parseCatalogIndexImage()
  — image reference parsing utilities

kube-client.ts (KubeClient class):
- createNamespace() — 409-safe namespace creation
- deleteNamespaceIfExists() — 404-safe deletion wrapping
  deleteNamespaceAndWait()

runtime-config.ts re-exports ImageRef, imageRefToString, and
parseCatalogIndexImage from helper.ts so existing callers are
unaffected.

runtime-deploy.ts now imports all shared utilities instead of
defining them locally, reducing file size by ~100 lines.

Assisted-by: OpenCode
- Guard against undefined host in RDS/Azure DB test loops by skipping
  individual DB versions when their host env var is not set, instead of
  crashing with a TypeError in clearDatabase/configurePostgresCredentials
- Fix RELEASE_NAME default mismatch in verify-schema-mode.spec.ts
  ('developer-hub' -> 'rhdh') to match runtime-config.ts and kube-client.ts
- Use resolveInstallMethod() in getRhdhDeploymentName() instead of
  duplicating install method detection logic
- Rewrite updateConfigMapTitle as a thin wrapper around patchAppConfig,
  eliminating ~65 lines of duplicated read-modify-write ConfigMap logic
- Extract deployment env var manipulation from postgres-config.ts into
  KubeClient methods (removeContainerEnvVars, addContainerEnvVarsFromSecret)
  so postgres-config.ts stays at the 'what' level while KubeClient handles
  'how' to patch deployments

Assisted-by: OpenCode
- Extract BACKSTAGE_CR_API_VERSION constant in runtime-config.ts and
  import it in runtime-deploy.ts, so there is a single place to update
  when the CRD version bumps
- Document the workers:1 assumption on the module-level deployed flag
  in runtime-deploy.ts to help future readers
- Add comment explaining why the operator path uses a computed route URL
  rather than cluster discovery (deterministic naming convention)

Assisted-by: OpenCode
Each runtime spec now calls ensureRuntimeDeployed() in its own
beforeAll instead of relying on alphabetical file discovery order
(configuration-test/ running before external-database/ and
plugin-division-mode-schema/). The call is idempotent — when tests
run in order it detects the existing ready deployment and returns
immediately. When a spec is run standalone via --grep, it deploys
RHDH first.

This removes the hidden ordering dependency that the old explicit
`dependencies: [SHOWCASE_RUNTIME_DB]` used to make visible.

Assisted-by: OpenCode
KubeClient now uses loadFromDefault() so these env vars are no
longer consumed by runtime-deploy.ts. Remove them from the header
to keep the documentation accurate.

Assisted-by: OpenCode
Change config.image from a plain {registry, repository, tag} object
to ImageRef which includes a separator field (':' for tags, '@' for
digests). This fixes digest-pinned images (e.g. repo@sha256:...)
being incorrectly joined with ':' in generateBackstageCR().

The same ImageRef/imageRefToString() is already used for
catalogIndex — this unifies both image references under the same
type and construction logic.

Assisted-by: OpenCode
Centralize the backstage-backend container name as a named constant
in kube-client.ts alongside getRhdhDeploymentName(). Replace
hardcoded strings in runtime-config.ts, postgres-config.ts, and
schema-mode-setup.ts.

rhdh-deployment.ts (auth-providers) also uses the same string but
is left for a future PR to avoid expanding scope.

Assisted-by: OpenCode
Extract the duplicated `tag.startsWith("sha256:") ? "@" : ":"
separator logic into a buildImageRef() helper in helper.ts. Used by
both runtime-config.ts and rhdh-deployment.ts to construct ImageRef
from individual registry/repository/tag env vars.

Assisted-by: OpenCode
Deduplicate the overlapping POSTGRES_* env var key lists across
postgres-config.ts and schema-mode-setup.ts into a shared
POSTGRES_ENV_KEYS constant. The external-DB function extends it
with PGSSLMODE and NODE_EXTRA_CA_CERTS via POSTGRES_CRED_ENV_KEYS.

Assisted-by: OpenCode
Move configureSchemaMode() and the default DB user/password constants
from runtime-deploy.ts to schema-mode-db.ts where the rest of the
schema-mode database utilities live. runtime-deploy.ts imports and
calls the function — it no longer owns any schema-mode-specific logic.

Assisted-by: OpenCode
updateConfigMapTitle was a thin wrapper around patchAppConfig with
an unused _configMapName parameter and a single caller. Inline the
title-update logic into config-map.spec.ts and remove the method
from KubeClient.

Assisted-by: OpenCode
Rename POSTGRES_CRED_ENV_KEYS to postgresCredEnvKeys — private
(non-exported) const must be camelCase per the eslint
@typescript-eslint/naming-convention rule.

Assisted-by: OpenCode
The placeholder postgres-cred secret doesn't contain a POSTGRES_DB key,
causing CreateContainerConfigError when prepareForExternalDatabase()
adds a secretKeyRef for it. External DB tests don't need POSTGRES_DB —
Backstage auto-creates per-plugin databases when it's unset.

Schema-mode tests are unaffected; they manage env vars via their own
configureSchemaMode() / schema-mode-setup.ts mechanism.

Assisted-by: OpenCode
After rebasing onto main, the kube-client.ts monolith was split into
kube-client/ directory modules. Add methods and constants that our PR
introduced to the new modular structure:

- BACKSTAGE_BACKEND_CONTAINER constant
- patchAppConfig, jsonPatchDeployment, restartDeploymentWithRetry
- removeContainerEnvVars, addContainerEnvVarsFromSecret
- waitForBackstageCrd, createConfigMap, deleteNamespaceIfExists, createNamespace
- getRhdhDeploymentName: use resolveInstallMethod() for INSTALL_METHOD env var

Fix all oxlint violations to comply with strict + pedantic linting:
- Replace || with ?? for nullish coalescing
- Add explicit nullish/empty checks for strict-boolean-expressions
- Fix setTimeout in Promise executors for strict-void-return
- Move inline comments to separate lines
- Add type annotations and safe type assertions
- Swap negated conditions to positive form

Assisted-by: OpenCode
…states

The checkWaitingContainerState function had inverted logic — it treated
transient states like PodInitializing and ContainerCreating as failures
while silently ignoring actual failure states like CrashLoopBackOff and
ImagePullBackOff.

This caused the runtime tests to fail immediately during operator
deployment because the init container (install-dynamic-plugins) triggers
PodInitializing state, which was incorrectly flagged as a pod failure.

Assisted-by: OpenCode
Runtime tests that navigate to the RHDH UI establish WebSocket
connections (event-stream). When the test body completes, Playwright's
page fixture teardown hangs for up to 10 minutes waiting for the browser
context to close because the WebSocket connections are never dropped.

Fix: navigate to about:blank at the end of each test that uses the page
fixture to close all active connections before Playwright tears down the
context.

Also add a 10-minute project-level timeout for the SHOWCASE_RUNTIME
project — runtime tests restart the RHDH deployment (ConfigMap changes,
external DB reconfiguration, schema-mode setup) which takes 60-90s per
restart.

Assisted-by: OpenCode
@zdrapela
zdrapela force-pushed the fix/showcase-runtime-test-ordering branch from daeefd1 to ac701b4 Compare June 26, 2026 10:01
@zdrapela

Copy link
Copy Markdown
Member Author

/test e2e-ocp-operator-nightly
/test e2e-ocp-helm-nightly

@github-actions

Copy link
Copy Markdown
Contributor

Image was built and published successfully. It is available at:

@openshift-ci

openshift-ci Bot commented Jun 26, 2026

Copy link
Copy Markdown

@zdrapela: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-ocp-operator-auth-providers-nightly 5c292df link false /test e2e-ocp-operator-auth-providers-nightly
ci/prow/e2e-ocp-operator-nightly ac701b4 link false /test e2e-ocp-operator-nightly

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-ci openshift-ci Bot added the lgtm label Jun 26, 2026
@openshift-merge-bot
openshift-merge-bot Bot merged commit 036635a into redhat-developer:main Jun 26, 2026
23 of 24 checks passed
@sonarqubecloud

Copy link
Copy Markdown

schultzp2020 added a commit that referenced this pull request Jun 26, 2026
Reconcile e2e hardening with main's runtime/kube-client refactor and fix
worker-scoped rhdhPage fixture crash when testInfo.titlePath is unset.

Co-authored-by: Cursor <cursoragent@cursor.com>
Zaperex pushed a commit to Zaperex/rhdh that referenced this pull request Jun 29, 2026
…er#4809)

* fix(e2e): consolidate SHOWCASE_RUNTIME_DB into SHOWCASE_RUNTIME project

Merge the SHOWCASE_RUNTIME_DB Playwright project into SHOWCASE_RUNTIME
to simplify runtime test execution. All runtime tests now run
sequentially in a single project (workers: 1) with no inter-project
dependencies.

Key changes:
- Deploy runtime with internal PostgreSQL (Helm sub-chart / operator-
  managed) instead of external Crunchy DB
- External DB tests (RDS, Azure) switch to external DB at runtime via
  prepareForExternalDatabase() which patches ConfigMap + adds env vars
- Operator uses separate rhdh-runtime-config secret for RHDH_RUNTIME_URL
  to avoid POSTGRES_* env var conflicts with internal DB
- schema-mode-env.sh and schema-mode-setup.ts support operator-specific
  service/secret naming (backstage-psql-*) and POSTGRESQL_ADMIN_PASSWORD
  preservation
- SSL connection conditionally applied (skip for internal DB, enable for
  external DB)
- Added waitForRuntimeDeploymentReady() for config-map tests
- Added restartWithRetry() for operator reconciliation resilience
- Removed old resource files (values-showcase-postgres.yaml,
  rds-app-config.yaml, rhdh-start-runtime.yaml)

Assisted-by: OpenCode

* refactor(e2e): extract shared utilities and eliminate duplication

- Extract resolveInstallMethod(), base64Encode/Decode to helper.ts
- Deduplicate AppConfigYaml interface into runtime-config.ts
- Add patchAppConfig(), restartDeploymentWithRetry(), jsonPatchDeployment()
  to KubeClient — eliminates repeated ConfigMap patch + deployment restart
  patterns across postgres-config.ts and schema-mode-setup.ts
- Export getKubeApiErrorMessage, run(), discoverRouterBase() for reuse
- Deduplicate getDeploymentName() via getRhdhDeploymentName()
- Handle @sha256: digest refs in parseCatalogIndexImage()
- Type BackstageCR return value

Assisted-by: OpenCode

* fix(e2e): address review findings

- Fix parseCatalogIndexImage digest reconstruction: add ImageRef.separator
  field (':' for tags, '@' for digests) and imageRefToString() helper so
  downstream consumers reconstruct refs correctly
- Fix patchAppConfig no-op detection: compare YAML before/after mutator
  to skip unnecessary ConfigMap writes and misleading log messages
- Fix removeSchemaModePatchedEnvVars: add explicit backstageIdx === -1
  guard with warning log (consistent with ensurePostgresCredEnvVars)

Assisted-by: OpenCode

* refactor(e2e): unify KubeClient with rhdh-deployment.ts patterns

- KubeClient constructor: switch from loadFromOptions() with explicit
  K8S_CLUSTER_URL/K8S_CLUSTER_TOKEN env vars to loadFromDefault() which
  reads the kubeconfig file (set by oc login / kubectl config). This
  aligns with how rhdh-deployment.ts (auth-providers) connects.
- Extract waitForBackstageCrd() as standalone function in kube-client.ts,
  used by both runtime-deploy.ts and rhdh-deployment.ts — eliminates
  duplicated CRD polling loops.
- rhdh-deployment.ts: adopt base64Encode() from helper.ts,
  getKubeApiErrorMessage() from kube-client.ts for safe error logging.

Assisted-by: OpenCode

* refactor(e2e): standardize on js-yaml, drop yaml package

Migrate rhdh-deployment.ts from the `yaml` npm package to `js-yaml`
which is already used by kube-client.ts and runtime-config.ts. The two
packages have incompatible APIs (`yaml.parse/stringify` vs
`yaml.load/dump`) despite being imported under the same alias,
creating a copy-paste trap for developers.

Changes:
- rhdh-deployment.ts: yaml.parse() -> yaml.load(),
  yaml.stringify() -> yaml.dump()
- package.json: remove direct `yaml` dependency from e2e-tests

Assisted-by: OpenCode

* refactor(e2e): adopt shared helpers and fix createConfigMap typo

- Replace 8 inline Buffer.from() base64 calls with base64Encode/Decode
  from helper.ts (keycloak.ts, api-helper.ts, annotator.spec.ts,
  scaffolder-relation-processor.spec.ts)
- Fix method name typo: createCongifmap -> createConfigMap in KubeClient
- Use kubeClient.createConfigMap() in runtime-deploy.ts instead of
  inline coreV1Api.createNamespacedConfigMap() calls

Assisted-by: OpenCode

* refactor(e2e): extract shared utilities from runtime-*.ts

Move reusable utilities out of runtime-deploy.ts and
runtime-config.ts into shared modules:

helper.ts:
- run() — shell command execution with stdout/stderr capture
- discoverRouterBase() — OpenShift cluster router base discovery
- ImageRef interface, imageRefToString(), parseCatalogIndexImage()
  — image reference parsing utilities

kube-client.ts (KubeClient class):
- createNamespace() — 409-safe namespace creation
- deleteNamespaceIfExists() — 404-safe deletion wrapping
  deleteNamespaceAndWait()

runtime-config.ts re-exports ImageRef, imageRefToString, and
parseCatalogIndexImage from helper.ts so existing callers are
unaffected.

runtime-deploy.ts now imports all shared utilities instead of
defining them locally, reducing file size by ~100 lines.

Assisted-by: OpenCode

* fix(e2e): address review findings in runtime test refactor

- Guard against undefined host in RDS/Azure DB test loops by skipping
  individual DB versions when their host env var is not set, instead of
  crashing with a TypeError in clearDatabase/configurePostgresCredentials
- Fix RELEASE_NAME default mismatch in verify-schema-mode.spec.ts
  ('developer-hub' -> 'rhdh') to match runtime-config.ts and kube-client.ts
- Use resolveInstallMethod() in getRhdhDeploymentName() instead of
  duplicating install method detection logic
- Rewrite updateConfigMapTitle as a thin wrapper around patchAppConfig,
  eliminating ~65 lines of duplicated read-modify-write ConfigMap logic
- Extract deployment env var manipulation from postgres-config.ts into
  KubeClient methods (removeContainerEnvVars, addContainerEnvVarsFromSecret)
  so postgres-config.ts stays at the 'what' level while KubeClient handles
  'how' to patch deployments

Assisted-by: OpenCode

* fix(e2e): extract API version constant and clarify assumptions

- Extract BACKSTAGE_CR_API_VERSION constant in runtime-config.ts and
  import it in runtime-deploy.ts, so there is a single place to update
  when the CRD version bumps
- Document the workers:1 assumption on the module-level deployed flag
  in runtime-deploy.ts to help future readers
- Add comment explaining why the operator path uses a computed route URL
  rather than cluster discovery (deterministic naming convention)

Assisted-by: OpenCode

* fix(e2e): call ensureRuntimeDeployed in every runtime spec

Each runtime spec now calls ensureRuntimeDeployed() in its own
beforeAll instead of relying on alphabetical file discovery order
(configuration-test/ running before external-database/ and
plugin-division-mode-schema/). The call is idempotent — when tests
run in order it detects the existing ready deployment and returns
immediately. When a spec is run standalone via --grep, it deploys
RHDH first.

This removes the hidden ordering dependency that the old explicit
`dependencies: [SHOWCASE_RUNTIME_DB]` used to make visible.

Assisted-by: OpenCode

* fix(e2e): drop stale K8S_CLUSTER_URL/TOKEN from header comment

KubeClient now uses loadFromDefault() so these env vars are no
longer consumed by runtime-deploy.ts. Remove them from the header
to keep the documentation accurate.

Assisted-by: OpenCode

* refactor(e2e): use ImageRef for main container image

Change config.image from a plain {registry, repository, tag} object
to ImageRef which includes a separator field (':' for tags, '@' for
digests). This fixes digest-pinned images (e.g. repo@sha256:...)
being incorrectly joined with ':' in generateBackstageCR().

The same ImageRef/imageRefToString() is already used for
catalogIndex — this unifies both image references under the same
type and construction logic.

Assisted-by: OpenCode

* refactor(e2e): extract BACKSTAGE_BACKEND_CONTAINER constant

Centralize the backstage-backend container name as a named constant
in kube-client.ts alongside getRhdhDeploymentName(). Replace
hardcoded strings in runtime-config.ts, postgres-config.ts, and
schema-mode-setup.ts.

rhdh-deployment.ts (auth-providers) also uses the same string but
is left for a future PR to avoid expanding scope.

Assisted-by: OpenCode

* refactor(e2e): extract buildImageRef helper for separator detection

Extract the duplicated `tag.startsWith("sha256:") ? "@" : ":"
separator logic into a buildImageRef() helper in helper.ts. Used by
both runtime-config.ts and rhdh-deployment.ts to construct ImageRef
from individual registry/repository/tag env vars.

Assisted-by: OpenCode

* refactor(e2e): extract shared POSTGRES_ENV_KEYS constant

Deduplicate the overlapping POSTGRES_* env var key lists across
postgres-config.ts and schema-mode-setup.ts into a shared
POSTGRES_ENV_KEYS constant. The external-DB function extends it
with PGSSLMODE and NODE_EXTRA_CA_CERTS via POSTGRES_CRED_ENV_KEYS.

Assisted-by: OpenCode

* refactor(e2e): move schema-mode config to schema-mode-db.ts

Move configureSchemaMode() and the default DB user/password constants
from runtime-deploy.ts to schema-mode-db.ts where the rest of the
schema-mode database utilities live. runtime-deploy.ts imports and
calls the function — it no longer owns any schema-mode-specific logic.

Assisted-by: OpenCode

* refactor(e2e): remove updateConfigMapTitle, use patchAppConfig directly

updateConfigMapTitle was a thin wrapper around patchAppConfig with
an unused _configMapName parameter and a single caller. Inline the
title-update logic into config-map.spec.ts and remove the method
from KubeClient.

Assisted-by: OpenCode

* fix(e2e): fix naming convention lint error for private constant

Rename POSTGRES_CRED_ENV_KEYS to postgresCredEnvKeys — private
(non-exported) const must be camelCase per the eslint
@typescript-eslint/naming-convention rule.

Assisted-by: OpenCode

* fix(e2e): don't inject POSTGRES_DB env var in external DB tests

The placeholder postgres-cred secret doesn't contain a POSTGRES_DB key,
causing CreateContainerConfigError when prepareForExternalDatabase()
adds a secretKeyRef for it. External DB tests don't need POSTGRES_DB —
Backstage auto-creates per-plugin databases when it's unset.

Schema-mode tests are unaffected; they manage env vars via their own
configureSchemaMode() / schema-mode-setup.ts mechanism.

Assisted-by: OpenCode

* fix(e2e): adapt new KubeClient methods to modular kube-client structure

After rebasing onto main, the kube-client.ts monolith was split into
kube-client/ directory modules. Add methods and constants that our PR
introduced to the new modular structure:

- BACKSTAGE_BACKEND_CONTAINER constant
- patchAppConfig, jsonPatchDeployment, restartDeploymentWithRetry
- removeContainerEnvVars, addContainerEnvVarsFromSecret
- waitForBackstageCrd, createConfigMap, deleteNamespaceIfExists, createNamespace
- getRhdhDeploymentName: use resolveInstallMethod() for INSTALL_METHOD env var

Fix all oxlint violations to comply with strict + pedantic linting:
- Replace || with ?? for nullish coalescing
- Add explicit nullish/empty checks for strict-boolean-expressions
- Fix setTimeout in Promise executors for strict-void-return
- Move inline comments to separate lines
- Add type annotations and safe type assertions
- Swap negated conditions to positive form

Assisted-by: OpenCode

* fix(e2e): fix inverted pod failure detection for transient container states

The checkWaitingContainerState function had inverted logic — it treated
transient states like PodInitializing and ContainerCreating as failures
while silently ignoring actual failure states like CrashLoopBackOff and
ImagePullBackOff.

This caused the runtime tests to fail immediately during operator
deployment because the init container (install-dynamic-plugins) triggers
PodInitializing state, which was incorrectly flagged as a pod failure.

Assisted-by: OpenCode

* fix(e2e): fix runtime test timeouts caused by WebSocket teardown hang

Runtime tests that navigate to the RHDH UI establish WebSocket
connections (event-stream). When the test body completes, Playwright's
page fixture teardown hangs for up to 10 minutes waiting for the browser
context to close because the WebSocket connections are never dropped.

Fix: navigate to about:blank at the end of each test that uses the page
fixture to close all active connections before Playwright tears down the
context.

Also add a 10-minute project-level timeout for the SHOWCASE_RUNTIME
project — runtime tests restart the RHDH deployment (ConfigMap changes,
external DB reconfiguration, schema-mode setup) which takes 60-90s per
restart.

Assisted-by: OpenCode
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants