docs: publish RHDH plugin quality requirements by support level (RHIDP-13497)

[gustavolira]

Summary

Publishes the canonical RHDH plugin quality requirements document, defining differentiated testing requirements per support level (GA, Tech-Preview, Community, Dev-Preview).

This document is referenced from the RHDH Plugin Ecosystem Workflow spreadsheet (row 5.0 — Quality) and replaces the Jira-only requirements that were difficult to find and reference.

Key content

• Requirements matrix with coverage minimums, quality gates, and enforcement levels per support level
• GA workspace E2E coverage table — verified 2026-07-24: 14/17 GA workspaces (82%) have E2E tests; 3 gaps remain (apiconnect, dynatrace-dql, scaffolder-backend-module-regex — all upstream/3rd-party owned)
• Resolved open questions: TP enforcement only at promotion, Community load test required, same frontend/backend requirements initially
• Document governance: owner, review cadence, change process
• Recent progress section: delivered infrastructure across all 3 repos
• Quality metrics dashboard with current baselines per support level

Why a repo document instead of Jira?

Per stakeholder feedback: quality requirements should be stable, visible, and easily linked — not buried in Jira comments. This document is versioned, reviewable via PR, and linkable from the Plugin Ecosystem Workflow.

Data sources

• Support levels: rhdh-plugin-export-overlays/workspaces/*/metadata/*.yaml
• E2E/smoke test presence: overlay repo workspace directories
• Unit test counts: rhdh-plugins workspace directories
• Coverage dashboards: Codecov rhdh-plugins

Test plan

• Document renders correctly on GitHub
• All workspace counts match overlay repo metadata
• Links to related docs resolve correctly
• Referenced Jira tickets (RHIDP-13497, RHIDP-15513, etc.) are valid

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.96%. Comparing base (d090bd1) to head (2759ebd).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5158      +/-   ##
==========================================
- Coverage   63.69%   59.96%   -3.74%     
==========================================
  Files         123      111      -12     
  Lines        2424     2198     -226     
  Branches      572      523      -49     
==========================================
- Hits         1544     1318     -226     
- Misses        878      879       +1     
+ Partials        2        1       -1     

Flag	Coverage Δ
rhdh	59.96% <ø> (-3.74%)	⬇️

Components	Coverage Δ
Backend plugins	∅ <ø> (∅)
Backend app	66.66% <ø> (ø)
Frontend app	58.89% <ø> (ø)
Plugin utils	∅ <ø> (∅)
Dynamic plugins utils	∅ <ø> (∅)

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d090bd1...2759ebd. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[rhdh-qodo-merge]

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 🔗 Cross-repo conflicts (1) 📜 Skill insights (0)

Context used

⚠️ Tickets: not configured — ticket URL found in PR but could not be fetched — check ticket provider credentials

✅ Compliance rules (platform): 46 rules

✅ Skills: e2e-verify-fix, e2e-diagnose-and-fix

✅ Cross-repo context

  Explored: repo: redhat-developer/rhdh-plugins (sha: 77b7f157)

  Not relevant to this PR: redhat-developer/rhdh-chart

  Not relevant to this PR: redhat-developer/rhdh-operator

  Not relevant to this PR: redhat-developer/rhdh-local

1. Non-portable JUnit parsing 🐞 Bug ☼ Reliability

Description

testing::_count_junit_failures uses GNU/PCRE-specific grep -oP with \K, which is unsupported by
BSD grep (macOS) and some minimal Linux images. When unsupported, it can yield UNKNOWN/misreported
failure counts (and may fail outright in stricter shells), affecting both regular Playwright runs
and the new plugin-sanity job reporting.

Code

.ci/pipelines/lib/testing.sh[R81-84]

+  local failures errors total
+  failures=$(grep -oP 'failures="\K[0-9]+' "${junit_file}" | head -n 1)
+  errors=$(grep -oP 'errors="\K[0-9]+' "${junit_file}" | head -n 1)
+  total=$((${failures:-0} + ${errors:-0}))

Relevance

⭐⭐⭐ High

Team previously accepted replacing GNU/PCRE grep -oP/\K due to BSD/macOS non-portability.

PR-#4561

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The new helper uses grep -oP/\K to extract XML attributes, which is a GNU/PCRE-only feature. The
repo’s own past accepted-bug guidance explicitly calls out this portability pitfall for grep -oP
and recommends POSIX/macOS-compatible alternatives.

.ci/pipelines/lib/testing.sh[64-90]
PR-#4561

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`testing::_count_junit_failures` parses JUnit XML with `grep -oP ... \K`, which is not portable (BSD grep lacks `-P`), and can result in UNKNOWN/misreported failure counts in local runs or alternative CI images.

### Issue Context
This helper is now used for both `testing::run_tests` and `testing::run_plugin_sanity_check`, so the portability gap affects more than one CI/test path.

### Fix Focus Areas
- .ci/pipelines/lib/testing.sh[64-90]

### Suggested fix
Replace the `grep -oP` usage with a portable approach, e.g.:
- `sed -nE` anchored on the `<testsuites ...>` element attributes, or
- `awk` attribute extraction, or
- a small `python3` snippet using `xml.etree.ElementTree` if Python is guaranteed in the CI image.

Ensure the logic still:
1) sums `failures + errors`, and
2) returns `UNKNOWN_FAILURE_COUNT` when the file is missing or when a non-zero Playwright exit produced a 0+0 total.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Unbounded oc logs download ✓ Resolved 🐞 Bug ➹ Performance

Description

testing::report_plugin_startup_failures downloads full current and previous logs for all matching
pods/containers without any --tail/--since bounds. ocp-nightly.sh calls it unconditionally, so
successful nightly runs still incur potentially large log transfers and API load.

Code

.ci/pipelines/lib/testing.sh[R254-260]

+  {
+    local pod
+    for pod in $(oc get pods -n "${namespace}" -o name 2> /dev/null | grep -E 'backstage|developer-hub' || true); do
+      # Current and previous (pre-crash) logs; either may not exist yet.
+      oc logs "${pod}" -n "${namespace}" --all-containers 2> /dev/null || true
+      oc logs "${pod}" -n "${namespace}" --all-containers -p 2> /dev/null || true
+    done

Relevance

⭐⭐⭐ High

Repo has accepted reducing/conditioning pod log collection to speed healthy runs; bounding oc logs
aligns.

PR-#4267

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The reporter loops pods and requests unbounded logs (oc logs without --tail/--since) for all
containers, both current and previous. The nightly job calls this reporter every time the
sanity-plugins check runs, regardless of whether the deployment/tests succeeded.

.ci/pipelines/lib/testing.sh[241-270]
.ci/pipelines/jobs/ocp-nightly.sh[70-88]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`testing::report_plugin_startup_failures` runs on every nightly job invocation and calls `oc logs` twice per pod (`current` and `-p` previous) with `--all-containers` and no size/time bounds. This can slow healthy runs and increase OpenShift API/log streaming load.

### Issue Context
The function’s output is filtered down to a small summary, but the full logs are still retrieved before filtering.

### Fix Focus Areas
- .ci/pipelines/lib/testing.sh[249-270]
- .ci/pipelines/jobs/ocp-nightly.sh[70-88]

### Suggested fix
Implement one (or more) of the following:
- Add bounds to log retrieval, e.g. `oc logs ... --since=30m --tail=5000` (choose values appropriate for startup-failure detection).
- Only fetch `-p` previous logs when the pod has restarted (CrashLoopBackOff / restartCount > 0).
- Gate the call in `ocp-nightly.sh` so it runs only when the sanity-plugins deployment/tests failed (to avoid cost on fully healthy runs).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Installer CLI version drift 🔗 Cross-repo conflict ≡ Correctness

Description

The new cluster-free plugin sanity workflow installs plugins via
@red-hat-developer-hub/cli-module-install-dynamic-plugins@0.2.0, but this repo already depends on
0.3.0 of that CLI from rhdh-plugins, which added support for the enabled field (preferred over
disabled). This version drift can make the sanity harness disagree with the installer behavior
shipped/expected by rhdh-plugins (e.g., if catalog/index configs start using enabled), reducing
the test’s cross-repo fidelity.

Code

e2e-tests/local-harness/populate.sh[R22-24]

# Pinned so local runs install the exact CLI version CI uses.
CLI_VERSION="0.2.0"

Relevance

⭐⭐ Medium

Pin was previously introduced/accepted to match CI; unclear if CI moved to 0.3.0 or pin is
intentional.

PR-#5005

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The PR pins the installer CLI to 0.2.0 in the local harness, while the PR repo itself depends on
0.3.0; in rhdh-plugins, the package is versioned 0.3.0 and its changelog/types show enabled was
added in 0.3.0 with defined precedence over disabled, meaning older pins risk schema/behavior
drift as configs evolve.

e2e-tests/local-harness/populate.sh[22-48]
packages/backend/package.json[75-80]
External repo: redhat-developer/rhdh-plugins, workspaces/install-dynamic-plugins/packages/install-dynamic-plugins/package.json [1-6]
External repo: redhat-developer/rhdh-plugins, workspaces/install-dynamic-plugins/packages/install-dynamic-plugins/CHANGELOG.md [1-8]
External repo: redhat-developer/rhdh-plugins, workspaces/install-dynamic-plugins/packages/install-dynamic-plugins/src/types.ts [31-48]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`e2e-tests/local-harness/populate.sh` pins `@red-hat-developer-hub/cli-module-install-dynamic-plugins` to `0.2.0`, while the repo already depends on `0.3.0` from `rhdh-plugins` (which introduces the `enabled` field and its precedence rules). This can cause the new plugin-sanity harness to behave differently than the installer version used elsewhere in RHDH / expected from `rhdh-plugins`.

## Issue Context
- `rhdh-plugins` v0.3.0 adds `enabled` (preferred) with backward compatibility for `disabled`.
- The plugin-sanity flow is intended to validate catalog-index composition against the “current backend line”, so it should use the same installer semantics as the shipped toolchain.

## Fix Focus Areas
- e2e-tests/local-harness/populate.sh[22-48]
- packages/backend/package.json[75-80]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[rhdh-qodo-merge]

PR Summary by Qodo

Add cluster-free plugin sanity check and publish plugin quality requirements doc

✨ Enhancement 🧪 Tests 📝 Documentation ⚙️ Configuration changes 🕐 40+ Minutes

AI Description

• Publishes docs/testing-requirements-matrix.md, a canonical per-support-level
 (GA/TP/Community/Dev-Preview) plugin quality requirements matrix, replacing Jira-only tracking.
• Adds a new cluster-free "plugin sanity check" ([RHIDP-13508](https://redhat.atlassian.net/browse/RHIDP-13508)) that boots packages/backend from
 source with every plugin declared by the catalog index and verifies the dynamic plugin loader
 actually loaded them all.
• Refactors the shared cluster-free harness webServer/config-args setup out of
 playwright.legacy-local.config.ts into a reusable local-harness-servers.ts module, with no
 behavior change to the existing harness.
• Extends CI (ocp-nightly.sh / testing.sh) with plugin-startup-failure log reporting and a new
 testing::run_plugin_sanity_check job step, plus JUnit-publishing/failure-counting helper
 extraction.
• Adds supporting scripts (catalog-index-refs.sh, populate-catalog-index.sh,
 plugin-sanity-excludes.txt), a dummy config overlay (app-config.plugin-sanity.yaml), and unit
 tests for the new parsing utilities.

Diagram

graph TD
  A["ocp-nightly.sh"] --> B["testing::run_plugin_sanity_check"] --> C["populate-catalog-index.sh"]
  C --> D["catalog-index-refs.sh"] --> E[("Catalog Index Image")]
  C --> F["dynamic-plugins-root"]
  B --> G["playwright.plugin-sanity.config.ts"] --> H["packages/backend (from source)"]
  H --> I{{"loaded-plugins API"}}
  G --> J["plugin-dynamic-loading.spec.ts"]
  J --> I
  F --> J
  subgraph Legend
    direction LR
    _svc([Process/Script]) ~~~ _db[(External Image)] ~~~ _ext{{API Endpoint}}
  end

Loading

High-Level Assessment

The PR's approach is sound: it reuses the existing cluster-free harness pattern (established in PR #5005) rather than introducing a new test infrastructure, extracts shared webServer setup to avoid duplication, and layers the new sanity check as an independent, non-blocking CI step consistent with how the cluster-based sanity check already works. Publishing the quality-requirements matrix as a versioned repo doc (vs. Jira) is a reasonable, low-risk documentation choice explicitly justified by stakeholder feedback; no better alternative was evident.

Files changed (22) +1468 / -86

Enhancement (9) +681 / -41

ocp-nightly.shWire plugin-startup-failure reporting and cluster-free sanity check into nightly job +14/-0

Wire plugin-startup-failure reporting and cluster-free sanity check into nightly job

• Adds a call to report plugin startup failures after the cluster-based sanity check, and invokes the new cluster-free plugin sanity check step without aborting the job on failure.

.ci/pipelines/jobs/ocp-nightly.sh

testing.shExtract JUnit helpers and add cluster-free plugin sanity check + failure log scanning +192/-33

Extract JUnit helpers and add cluster-free plugin sanity check + failure log scanning

• Extracts JUnit publish/failure-count logic into reusable functions, adds testing::report_plugin_startup_failures to scan pod logs, and adds testing::run_plugin_sanity_check to run the new cluster-free check end-to-end.

.ci/pipelines/lib/testing.sh

catalog-index-refs.shNew script to extract and filter catalog index plugin refs +79/-0

New script to extract and filter catalog index plugin refs

• Pulls dynamic-plugins.default.yaml from a catalog index OCI image via skopeo, lists declared package refs, and filters out documented known-failure patterns.

e2e-tests/local-harness/catalog-index-refs.sh

populate-catalog-index.shNew script to populate dynamic-plugins-root from the full catalog index +59/-0

New script to populate dynamic-plugins-root from the full catalog index

• Generates an install config enabling every package declared by the catalog index and delegates to populate.sh, recording an expectation breadcrumb file for the sanity spec.

e2e-tests/local-harness/populate-catalog-index.sh

populate.shSupport alternate install-config path for populate.sh +31/-6

Support alternate install-config path for populate.sh

• Adds an optional first argument to select a different dynamic-plugins install config, resolving relative paths against the caller's cwd.

e2e-tests/local-harness/populate.sh

playwright.plugin-sanity.config.tsNew Playwright config for the cluster-free plugin sanity check +54/-0

New Playwright config for the cluster-free plugin sanity check

• Defines a dedicated, browser-less Playwright config that boots the backend with the plugin-sanity overlay and runs the plugin-dynamic-loading spec with JUnit/HTML reporting.

e2e-tests/playwright.plugin-sanity.config.ts

entry-graph.tsRegister plugin-sanity global setup in entry graph +4/-2

Register plugin-sanity global setup in entry graph

• Renames the legacy global setup import and adds the new plugin-sanity global setup to the dependency entry graph.

e2e-tests/playwright/entry-graph.ts

plugin-sanity-global-setup.tsNew global setup enforcing catalog-index population for sanity check +29/-0

New global setup enforcing catalog-index population for sanity check

• Extends the shared populated-check with a catalog-index-specific breadcrumb check so a leftover curated install cannot silently pass.

e2e-tests/playwright/support/plugin-sanity-global-setup.ts

plugin-loader.tsNew plugin manifest and loaded-plugins parsing utilities +219/-0

New plugin manifest and loaded-plugins parsing utilities

• Adds helpers to scan installed plugin directories, classify frontend/backend roles, validate frontend bundle artifacts, and parse the loaded-plugins API response and catalog-index breadcrumb.

e2e-tests/playwright/utils/plugin-loader.ts

Refactor (3) +85 / -44

playwright.legacy-local.config.tsRefactor legacy-local config to use shared harness server helpers +13/-39

Refactor legacy-local config to use shared harness server helpers

• Replaces inlined backend webServer, config args, and PATH shim with imports from the new shared local-harness-servers module; no behavior change.

e2e-tests/playwright.legacy-local.config.ts

local-harness-global-setup.tsGeneralize dynamic-plugins-root guard for multiple harnesses +14/-5

Generalize dynamic-plugins-root guard for multiple harnesses

• Parameterizes the fail-fast guard with per-harness run/populate command hints and exposes a default export scoped to the legacy-local config.

e2e-tests/playwright/support/local-harness-global-setup.ts

local-harness-servers.tsNew shared module for cluster-free harness backend/webServer setup +58/-0

New shared module for cluster-free harness backend/webServer setup

• Extracts the backend webServer definition, config-args builder, and PATH shim shared by the legacy-local and plugin-sanity Playwright configs.

e2e-tests/playwright/support/local-harness-servers.ts

Tests (2) +217 / -0

plugin-dynamic-loading.spec.tsNew spec validating catalog index plugins load in the backend +155/-0

New spec validating catalog index plugins load in the backend

• Enumerates installed plugins, authenticates as guest, queries the loaded-plugins API, and asserts every installed plugin (backend and frontend bundles) was loaded successfully.

e2e-tests/playwright/e2e/plugin-dynamic-loading.spec.ts

catalog-index-expectation.test.tsAdd unit tests for catalog index expectation parsing +62/-0

Add unit tests for catalog index expectation parsing

• Covers parsing of the populate-catalog-index.sh breadcrumb file including digest refs and malformed-input error handling.

e2e-tests/unit/catalog-index-expectation.test.ts

Documentation (3) +364 / -0

local-e2e-harness.mdDocument populate.sh optional config argument and plugin sanity hook +7/-0

Document populate.sh optional config argument and plugin sanity hook

• Explains the new optional install-config argument to populate.sh and links to the plugin sanity check documentation.

docs/e2e-tests/local-e2e-harness.md

testing-requirements-matrix.mdPublish RHDH plugin quality requirements matrix by support level +330/-0

Publish RHDH plugin quality requirements matrix by support level

• New canonical document defining differentiated test coverage requirements, quality gates, current E2E coverage state, governance, and roadmap per plugin support level (GA/TP/Community/Dev-Preview).

docs/testing-requirements-matrix.md

README.mdDocument the cluster-free Plugin Sanity Check +27/-0

Document the cluster-free Plugin Sanity Check

• Adds a new section explaining how to run the plugin sanity check locally and in CI, and how exclusions/dummy config are handled.

e2e-tests/README.md

Other (5) +121 / -1

app-config.plugin-sanity.yamlNew config overlay with dummy values for plugin sanity backend boot +79/-0

New config overlay with dummy values for plugin sanity backend boot

• Provides dummy config entries (argocd, tech-radar, kubernetes, LDAP, MS Graph) so plugins that validate config at startup do not abort the standalone backend.

app-config.plugin-sanity.yaml

.gitignoreIgnore generated JUnit and plugin-sanity report artifacts +2/-0

Ignore generated JUnit and plugin-sanity report artifacts

• Adds patterns to ignore junit-results-*.xml and playwright-report-* directories generated by the new sanity check.

e2e-tests/.gitignore

plugin-sanity-excludes.txtNew excludes list for plugins that cannot boot standalone +35/-0

New excludes list for plugins that cannot boot standalone

• Documents regex patterns for orchestrator-backend and its loki module, which are excluded from the cluster-free sanity check with rationale.

e2e-tests/local-harness/plugin-sanity-excludes.txt

package.jsonAdd plugin-sanity yarn script +2/-1

Add plugin-sanity yarn script

• Adds a new script to run the plugin-sanity Playwright config.

e2e-tests/package.json

playwright.config.tsExclude new plugin-dynamic-loading spec from cluster-based projects +3/-0

Exclude new plugin-dynamic-loading spec from cluster-based projects

• Adds the new plugin-dynamic-loading.spec.ts to testIgnore lists across multiple cluster-based Playwright projects.

e2e-tests/playwright.config.ts

[github-actions]

Image was built and published successfully. It is available at:

• quay.io/rhdh-community/rhdh:pr-5158
• quay.io/rhdh-community/rhdh:pr-5158-2759ebdc

[openshift-ci]

@gustavolira: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-ocp-helm	2759ebd	link	true	/test e2e-ocp-helm

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[github-actions]

The container image build and publish workflows were skipped (either due to [skip-build] tag or no relevant changes with existing image).

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[hopehadfield]

/lgtm