[chore] Adding nightly tests against next RHDH tag

[OpinionatedHeron]

Description

Added a nightly workflow to RHDH Local in order to run tests against the next RHDH tag. Tried to ensure that it would test all release branches, even if new ones are created.

Which issue(s) does this PR fix or relate to

RHIDP-12133

PR acceptance criteria

• Tests updated and passing
• Documentation updated
• Built-in TechDocs updated if needed. Note that TechDocs changes may need to be reviewed by a Product Manager and/or Architect to ensure content accuracy, clarity, and alignment with user needs.

How to test changes / Special notes to the reviewer

Tested workflow in my own fork, all tests seem to have run as expected but there were some failures for 1.8 and 1.7 - mostly related to arm, so I think they are to be expected.

[rhdh-qodo-merge]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 RHIDP-12133 - Partially compliant Compliant requirements: Add a nightly GitHub Actions workflow for RHDH Local. Run tests against the unstable/next RHDH tag (to catch breakages before users hit them). Ensure the workflow covers all release branches (including newly created ones) without needing manual workflow updates. Non-compliant requirements: Requires further human verification: Confirm the scheduled cron time is correct for the intended timezone/maintenance window and that the workflow reliably triggers nightly in the target repo.

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🔒 Security concerns Supply chain: The workflow executes npx --yes dclint ., which can fetch and run unpinned code during CI. Consider pinning the dclint version (or installing via the repository lockfile) to reduce supply-chain exposure and improve reproducibility.

⚡ Recommended focus areas for review Action Version The workflow uses actions/checkout@v6, which is likely an invalid/nonexistent major version and would cause the workflow to fail at runtime. Consider switching to a known supported major version (and ideally pin to a SHA for supply-chain hardening). - uses: actions/checkout@v6 - name: Lint Supply Chain The lint step runs npx --yes dclint ., which may download and execute the latest package at runtime. This can lead to non-reproducible runs and increases supply-chain risk. Consider pinning the package/version (or using a lockfile-based install) to keep the workflow deterministic. - name: Lint run: | npx --yes dclint .

📚 Focus areas based on broader codebase context Reproducibility The workflow pins several tool versions, but CORPORATE_PROXY_IMAGE is set to docker.io/ubuntu/squid:latest (line 73), which can change over time and make nightly results non-deterministic. Consider pinning this image to a specific version/digest (and optionally documenting the update policy) to keep failures attributable to code changes rather than drifting dependencies. (Ref 6, Ref 1) env: # Using the next tag from the release branch RHDH_TAG: ${{ github.event.inputs.rhdh_tag || vars.NEXT_RHDH_TAG }} RHDH_IMAGE: quay.io/rhdh-community/rhdh:next DOCKER_COMPOSE_VERSION: v5.0.1 PODMAN_IMAGE: quay.io/podman/stable:v5 CORPORATE_PROXY_IMAGE: docker.io/ubuntu/squid:latest Reference reasoning: Existing YAML in the codebase commonly uses explicit, stable image tags for test tooling and components (e.g., a fixed scorecard-test version and explicitly tagged component images), which avoids unexpected behavior changes from floating tags like latest.

📄 References redhat-developer/rhdh-operator/pkg/model/testdata/rhdh-deployment.yaml [36-58] redhat-developer/rhdh-chart/charts/orchestrator-infra/values.yaml [37-41] redhat-developer/rhdh-chart/charts/orchestrator-software-templates-infra/values.yaml [141-163] redhat-developer/rhdh-chart/charts/orchestrator-infra/templates/tests/infra-test.yaml [0-2] redhat-developer/rhdh-chart/charts/orchestrator-software-templates-infra/templates/tests/infra-test.yaml [0-2] redhat-developer/rhdh-operator/bundle/rhdh/tests/scorecard/config.yaml [5-47] redhat-developer/rhdh-operator/bundle/backstage.io/tests/scorecard/config.yaml [5-47] redhat-developer/rhdh-operator/config/scorecard/patches/olm.config.yaml [1-40]

[rhdh-qodo-merge]

PR Type

Tests, Enhancement

---

Description

• Add nightly workflow to test RHDH Local against next RHDH tag

• Dynamically discover and test all release branches at runtime

• Test matrix includes multiple OS, container tools, and configurations

• Lint checks and comprehensive health verification included

---

File Walkthrough

Relevant files

Tests

nightly.yaml Create nightly test workflow for RHDH releases                      .github/workflows/nightly.yaml New nightly workflow scheduled daily and manually triggerable via workflow_dispatch Lint job validates code quality using dclint Dynamic branch discovery job finds main and release-* branches at runtime Comprehensive test matrix covering ubuntu-24.04/arm, docker/podman, 6 compose configurations, and user config variations Health checks include container engine version display, compose config validation, app startup verification, and detailed logging on failure +244/-0

---

[rhdh-qodo-merge]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
High-level	✅ Reduce the nightly test matrix Suggestion Impact: The workflow was changed to run against a smaller set of branches (main + latest 3 release branches) and the compose configurations list was reduced, which decreases the overall matrix size and nightly workload (though not via schedule-vs-manual conditional matrices as suggested). code diff: @@ -28,11 +28,19 @@ id: set-branches with: script: | - const { data } = await github.rest.repos.listBranches({ owner: context.repo.owner, repo: context.repo.repo, per_page: 100 }); - const names = data.map(b => b.name).filter(n => n === 'main' || /^release-\d+\.\d+$/.test(n)); - const sorted = names.length ? names.sort((a, b) => (a === 'main' ? -1 : b === 'main' ? 1 : b.localeCompare(a, undefined, { numeric: true }))) : ['main']; - core.setOutput('branches', JSON.stringify(sorted)); + const branches = await github.paginate(github.rest.repos.listBranches, { + owner: context.repo.owner, + repo: context.repo.repo, + }); + // Keep main and the latest 3 release branches (e.g. release-1.2, release-1.1, release-1.0) - to cover EOL branches and period between FF and GA of new releases + const topReleaseBranches = branches + .map(b => b.name) + .filter(name => /^release-\d+\.\d+$/.test(name)) + .sort((a, b) => b.localeCompare(a, undefined, { numeric: true })) + .slice(0, 3); + + core.setOutput('branches', JSON.stringify(['main', ...topReleaseBranches])); test: needs: discover-branches strategy: @@ -57,189 +65,32 @@ cliArgs: "-f compose.yaml -f compose-dynamic-plugins-root.yaml" - name: "orchestrator-workflow" cliArgs: "-f compose.yaml -f orchestrator/compose.yaml" - - name: "developer-lightspeed-minimal" + - name: "developer-lightspeed" cliArgs: "-f compose.yaml -f developer-lightspeed/compose.yaml" - - name: "developer-lightspeed" - cliArgs: "-f compose.yaml -f developer-lightspeed/compose-with-ollama.yaml" The test matrix is too large for a nightly run, leading to high costs and long execution times. It should be reduced for scheduled runs, with the full matrix reserved for manual triggers. Examples: .github/workflows/nightly.yaml [38-63] strategy: fail-fast: false matrix: branch: ${{ fromJson(needs.discover-branches.outputs.branches) }} os: - ubuntu-24.04 - ubuntu-24.04-arm tool: - docker - podman ... (clipped 16 lines) Solution Walkthrough: Before: jobs: test: strategy: fail-fast: false matrix: branch: ${{ fromJson(needs.discover-branches.outputs.branches) }} os: - ubuntu-24.04 - ubuntu-24.04-arm tool: - docker - podman userConfig: - "false" - "true" composeConfig: - name: "default" - ... (5 more configurations) After: jobs: test: strategy: fail-fast: false matrix: # For nightly schedule, run a smaller, representative set of tests branch: ${{ github.event_name == 'schedule' && '["main"]' || fromJson(needs.discover-branches.outputs.branches) }} os: ${{ github.event_name == 'schedule' && '["ubuntu-24.04"]' || '["ubuntu-24.04", "ubuntu-24.04-arm"]' }} tool: ${{ github.event_name == 'schedule' && '["docker"]' || '["docker", "podman"]' }} userConfig: ${{ github.event_name == 'schedule' && '["false"]' || '["false", "true"]' }} composeConfig: ${{ github.event_name == 'schedule' && fromJson(''[{"name": "default", "cliArgs": ""}]'') || fromJson(needs.discover-branches.outputs.composeConfigs) }} ... Suggestion importance[1-10]: 9 __ Why: The suggestion correctly identifies a massive test matrix that would be excessively costly and slow for a nightly run, proposing a practical solution to run a smaller, representative set on schedule.	High
Possible issue	Fix invalid docker-compose version Update the DOCKER_COMPOSE_VERSION from the invalid v5.0.1 to a valid version like v2.27.0 to prevent the download step from failing. .github/workflows/nightly.yaml [71] -DOCKER_COMPOSE_VERSION: v5.0.1 +DOCKER_COMPOSE_VERSION: v2.27.0 Apply / Chat Suggestion importance[1-10]: 9 __ Why: The suggestion correctly identifies that v5.0.1 is an invalid version for docker/compose, which would cause the workflow to fail. This is a critical bug fix.	High
	✅ Add non-interactive install flag Suggestion Impact: The workflow no longer runs the `sudo apt-get install ca-certificates curl` command at all—the entire "Update Docker version" step containing it was removed/refactored into a reusable action, eliminating the interactive install risk the suggestion targeted. code diff: - - name: Update Docker version - run: | - # Add Docker's official GPG key: - sudo apt-get update - sudo apt-get install ca-certificates curl - sudo install -m 0755 -d /etc/apt/keyrings - sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc - sudo chmod a+r /etc/apt/keyrings/docker.asc - - # Add the repository to Apt sources: - echo \ - "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \ - $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | \ - sudo tee /etc/apt/sources.list.d/docker.list > /dev/null - sudo apt-get update - sudo apt-get install -y docker-ce \ Add the -y flag to the apt-get install command to ensure it runs non-interactively. .github/workflows/nightly.yaml [100] -sudo apt-get install ca-certificates curl +sudo apt-get install -y ca-certificates curl [Suggestion processed] Suggestion importance[1-10]: 7 __ Why: The suggestion correctly points out the missing -y flag, which is crucial for non-interactive execution in a CI environment to prevent the job from hanging.	Medium
General	Add workflow_dispatch input Define the rhdh_tag input under the workflow_dispatch trigger to make github.event.inputs.rhdh_tag available for manual runs. .github/workflows/nightly.yaml [3-4] on: workflow_dispatch: + inputs: + rhdh_tag: + description: 'Override RHDH tag' + required: false Apply / Chat Suggestion importance[1-10]: 8 __ Why: This suggestion correctly identifies that for github.event.inputs.rhdh_tag to be used, it must be defined as an input for the workflow_dispatch trigger, fixing a potential runtime error.	Medium
Update

[rm3l]

/agentic_review

[rhdh-qodo-merge]

Persistent review updated to latest commit d513065

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[rm3l]

/agentic_review

[rhdh-qodo-merge]

Persistent review updated to latest commit 3bd6382

[rm3l]

/cherry-pick release-1.10

[openshift-cherrypick-robot]

@rm3l: once the present PR merges, I will cherry-pick it on top of release-1.10 in a new PR and assign it to you.

Details

In response to this:

/cherry-pick release-1.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[openshift-cherrypick-robot]

@rm3l: new pull request created: #208

Details

In response to this:

/cherry-pick release-1.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-cherrypick-robot]

@OpinionatedHeron: #180 failed to apply on top of branch "release-1.9":

Applying: Creating Nightly test for RHDH Local
Applying: Updating tests due to failures
Applying: Updating github-script to use correct Node version
Applying: Updating matrix and rhdh_tag, and removing arm
Applying: Creating composite actions
Applying: Adding actions to fix test error
Applying: Updating based on test errors
Applying: Updating based on errors in nightly tests
Applying: Testing to fix failures
Applying: Using tags to decide what branches to test
Applying: Changes based on test errors
Applying: Adjusting due to test errors
Applying: Updating discover branches - checking 3 release branches
Applying: Updating action based on github security issue
Applying: Adding comment re number of release branches tested
Applying: Fixing flagged errors by qodo
Applying: Making fixes and removing unnecessary lines
Applying: Removing unnecessary parameters from test files
Applying: Making change to fix nightly test
Applying: Set appropriate tag for catalog index
Applying: Test to identify failure reason
Using index info to reconstruct a base tree...
M	default.env
Falling back to patching base and 3-way merge...
Auto-merging default.env
CONFLICT (content): Merge conflict in default.env
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0021 Test to identify failure reason

Details

In response to this:

/cherry-pick release-1.9
/cherry-pick release-1.8

Re-trying cherry pick based on bot feedback

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-cherrypick-robot]

@OpinionatedHeron: #180 failed to apply on top of branch "release-1.8":

Applying: Creating Nightly test for RHDH Local
Applying: Updating tests due to failures
Applying: Updating github-script to use correct Node version
Applying: Updating matrix and rhdh_tag, and removing arm
Using index info to reconstruct a base tree...
M	.github/workflows/test.yml
Falling back to patching base and 3-way merge...
Auto-merging .github/workflows/test.yml
CONFLICT (content): Merge conflict in .github/workflows/test.yml
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0004 Updating matrix and rhdh_tag, and removing arm

Details

In response to this:

/cherry-pick release-1.9
/cherry-pick release-1.8

Re-trying cherry pick based on bot feedback

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.