fix(ci): let the runtime specs deploy before global-setup polls their route

[gustavolira]

Problem

showcase-runtime fails in globalSetup, before a single test runs:

Error: expect(received).toBe(expected)
Expected: true
Received: false
Call Log:
- Timeout 120000ms exceeded while waiting on the predicate
   at ../utils/wait-for-rhdh-ready.ts:31
    at waitForRhdhReady (playwright/utils/wait-for-rhdh-ready.ts:31:6)
    at globalSetup (playwright/global-setup.ts:32:11)

Root cause — a deadlock, not a slow deployment

The runtime specs create their own namespace and deployment: config-map.spec.ts calls ensureRuntimeDeployed() in beforeAll. global-setup.ts is written for exactly that, and documents it:

 * - BASE_URL set → wait for that instance (CI or pre-deployed cluster)
 * - BASE_URL unset + RUNTIME_AUTO_DEPLOY=true → deploy showcase-runtime, then wait

CI never reached the second branch. testing::run_tests exports its optional url argument as BASE_URL, and both runtime callers passed the route:

testing::run_tests "${RELEASE_NAME}" "${NAME_SPACE_RUNTIME}" "${PW_PROJECT_SHOWCASE_RUNTIME}" "${runtime_url}"

So BASE_URL was always pre-set. The deploy branch was skipped, and globalSetup polled a route that nothing had created yet. It timed out after 120s, which meant the tests that would have done the deploying never ran.

RUNTIME_AUTO_DEPLOY is referenced only in global-setup.ts and set by no CI script, so that branch was dead code.

Fix

Stop passing the URL and set RUNTIME_AUTO_DEPLOY=true, so the documented path does the work. ensureRuntimeDeployed() sets BASE_URL itself once the route exists, and already waits for the deployment (waitForDeploymentReady, 600s) — so the readiness gate is not lost, it just happens after the thing exists.

Both callers are affected and both are fixed:

• jobs/ocp-nightly.sh (helm)
• jobs/ocp-operator.sh (operator)

Dropping the hard-coded routes is safe because runtime-deploy.ts already branches on the install method and returns the matching URL — <release>-developer-hub-<ns> for helm, backstage-<release>-<ns> for the operator, byte-identical to the strings removed here.

Verification

shellcheck and .ci prettier clean.

Honest caveat: this is not reproducible outside CI, so it was verified by reading the call path (run_tests → BASE_URL → global-setup → runtime-deploy), not by running it. Worth watching the first nightly after merge.

Side note, not fixed here

When runtime tests fail, no pod logs are collected — run_runtime_config_change_tests calls testing::run_tests directly, while testing::check_and_test is what wraps failures with save_all_pod_logs. That is why the failing run's showcase-runtime/ artifacts contain only the report and a 112-byte junit. Worth a follow-up.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[rhdh-qodo-merge]

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

Context used

✅ Compliance rules (platform): 46 rules

✅ Skills: e2e-verify-fix, e2e-diagnose-and-fix

✅ Cross-repo context

  Not relevant to this PR: redhat-developer/rhdh-plugins

  Not relevant to this PR: redhat-developer/rhdh-chart

  Not relevant to this PR: redhat-developer/rhdh-operator

  Not relevant to this PR: redhat-developer/rhdh-local

1. BASE_URL not set on reuse 🐞 Bug ≡ Correctness

Description

The jobs now set RUNTIME_AUTO_DEPLOY=true and call testing::run_tests without a URL, which exports
BASE_URL as an empty string and triggers global-setup.ts to call ensureRuntimeDeployed(). If a
runtime deployment already exists and is ready, ensureRuntimeDeployed() returns early without
setting process.env.BASE_URL, so Playwright starts with an empty baseURL and globalSetup also exits
without waiting on /healthcheck.

Code

.ci/pipelines/jobs/ocp-nightly.sh[R65-67]

  export INSTALL_METHOD="helm"
-  local runtime_url="https://${RELEASE_NAME}-developer-hub-${NAME_SPACE_RUNTIME}.${K8S_CLUSTER_ROUTER_BASE}"
-  testing::run_tests "${RELEASE_NAME}" "${NAME_SPACE_RUNTIME}" "${PW_PROJECT_SHOWCASE_RUNTIME}" "${runtime_url}" || true
+  export RUNTIME_AUTO_DEPLOY="true"
+  testing::run_tests "${RELEASE_NAME}" "${NAME_SPACE_RUNTIME}" "${PW_PROJECT_SHOWCASE_RUNTIME}" || true

Relevance

⭐⭐⭐ High

Team accepts CI/E2E correctness fixes; prior readiness/globalSetup changes merged in #5004; runtime
env correctness fixes accepted in #4288.

PR-#5004
PR-#4288
PR-#4809

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The PR change makes BASE_URL intentionally empty (by omitting the URL arg) while enabling
auto-deploy; global-setup will call ensureRuntimeDeployed() in that case. But
ensureRuntimeDeployed() has an early-return path when the deployment already exists and is ready
that never assigns BASE_URL, and global-setup/playwright config both depend on BASE_URL being set
for readiness polling and baseURL configuration.

.ci/pipelines/jobs/ocp-nightly.sh[59-68]
.ci/pipelines/jobs/ocp-operator.sh[80-85]
.ci/pipelines/lib/testing.sh[39-64]
e2e-tests/playwright/global-setup.ts[14-26]
e2e-tests/playwright/utils/runtime-deploy.ts[291-335]
e2e-tests/playwright.config.ts[38-64]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
CI now intentionally omits the runtime URL and enables `RUNTIME_AUTO_DEPLOY=true`, relying on Playwright `global-setup.ts` + `ensureRuntimeDeployed()` to populate `BASE_URL`. However, `ensureRuntimeDeployed()` has a fast-path where it detects an existing ready Deployment and returns without setting `process.env.BASE_URL`. In that scenario, `global-setup.ts` will subsequently see `BASE_URL` still empty and return, and Playwright will run with `baseURL` unset.

This can happen on CI re-runs or clusters where the `showcase-runtime` namespace was not fully cleaned up between job executions.

### Issue Context
* `testing::run_tests` exports `BASE_URL` from its optional URL arg, which is now omitted (so it becomes `""`).
* `global-setup.ts` deploys only when `BASE_URL` is empty and `RUNTIME_AUTO_DEPLOY` is true.
* `ensureRuntimeDeployed()` returns early when it finds an existing ready deployment, but does not set `BASE_URL` in that branch.
* Playwright config uses `process.env.BASE_URL` as `use.baseURL`.

### Fix Focus Areas
- e2e-tests/playwright/utils/runtime-deploy.ts[291-356]

### Implementation notes
1. In the `ready >= 1` early-return branch in `ensureRuntimeDeployed()`, if `process.env.BASE_URL` is empty/undefined, compute and set it:
  * For `operator`: deterministic URL is `https://backstage-${releaseName}-${namespace}.${routerBase}`.
  * For `helm`: prefer querying the Route host from the cluster (similar logic to `deployWithHelm()`), and fall back to computed `https://${routeName}-${namespace}.${routerBase}`.
2. After setting `BASE_URL`, keep existing schema-mode env configuration behavior.
3. Optionally add a small log line when `BASE_URL` is populated from an existing deployment to aid debugging.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[rhdh-qodo-merge]

PR Summary by Qodo

Fix CI deadlock by enabling runtime auto-deploy before Playwright globalSetup waits

🐞 Bug fix ⚙️ Configuration changes 🕐 10-20 Minutes

AI Description

• Stop passing a runtime URL so Playwright globalSetup can run the deploy path.
• Export RUNTIME_AUTO_DEPLOY=true in helm and operator runtime CI jobs.
• Document why BASE_URL must remain unset for runtime specs to deploy successfully.

Diagram

graph TD
  A[".ci jobs (nightly/operator)"] --> B["testing::run_tests"] --> C["Env: BASE_URL unset + RUNTIME_AUTO_DEPLOY=true"] --> D["Playwright global-setup.ts"] --> E["ensureRuntimeDeployed()"] --> F["Runtime deployed + BASE_URL set"]
  A --> G["No runtime_url arg passed"]

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Make global-setup prefer RUNTIME_AUTO_DEPLOY over BASE_URL

• ➕ More resilient to accidental BASE_URL being set by wrappers
• ➕ Centralizes the deploy vs. wait decision in one place
• ➖ Risk of deploying even when a real pre-provisioned BASE_URL is intended
• ➖ Changes test harness semantics beyond CI scripts

2. Change testing::run_tests to only export BASE_URL when explicitly requested

• ➕ Prevents similar deadlocks across other callers
• ➕ Keeps job scripts simpler and reduces footguns
• ➖ Potentially breaking change for existing callers relying on current behavior
• ➖ Requires auditing other pipelines and documentation

Recommendation: The PR’s approach (stop passing the URL and explicitly enabling RUNTIME_AUTO_DEPLOY) is the safest localized fix: it matches the documented global-setup contract and avoids changing shared test harness semantics. Consider the run_tests export behavior as a follow-up hardening, but it’s broader in blast radius.

Files changed (2) +15 / -5

Bug fix (2) +15 / -5

ocp-nightly.shEnable runtime auto-deploy and stop pre-setting BASE_URL +9/-3

Enable runtime auto-deploy and stop pre-setting BASE_URL

• Removes the hard-coded runtime route argument passed to testing::run_tests so BASE_URL is not pre-set. Exports RUNTIME_AUTO_DEPLOY=true to ensure Playwright global setup follows the deploy-then-wait path, with an explanatory comment documenting the prior deadlock.

.ci/pipelines/jobs/ocp-nightly.sh

ocp-operator.shAlign operator runtime tests with auto-deploy globalSetup contract +6/-2

Align operator runtime tests with auto-deploy globalSetup contract

• Removes the operator-specific hard-coded runtime route passed to testing::run_tests and instead enables RUNTIME_AUTO_DEPLOY=true. Adds a short note referencing the nightly job rationale to prevent BASE_URL from causing global-setup to poll a non-existent route.

.ci/pipelines/jobs/ocp-operator.sh

[github-actions]

Image was built and published successfully. It is available at:

• quay.io/rhdh-community/rhdh:pr-5136
• quay.io/rhdh-community/rhdh:pr-5136-f97abd76

[albarbaro]

/lgtm

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 56.03%. Comparing base (5992bdd) to head (f97abd7).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #5136   +/-   ##
=======================================
  Coverage   56.03%   56.03%           
=======================================
  Files         123      123           
  Lines        2404     2404           
  Branches      553      553           
=======================================
  Hits         1347     1347           
  Misses       1051     1051           
  Partials        6        6           

Flag	Coverage Δ		*Carryforward flag
rhdh	56.03% <ø> (ø)		Carriedforward from 5992bdd

*This pull request uses carry forward flags. Click here to find out more.

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5992bdd...f97abd7. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.