Skip to content

README.md

Latest commit

 

History

History
117 lines (82 loc) · 8.9 KB

README.md

File metadata and controls

117 lines (82 loc) · 8.9 KB

CI

CI is managed through the OpenShift CI system (Prow + ci-operator). The job configuration lives in openshift/release.

Jobs

Job Schedule Description
check-docs Pre-submit Checks markdown formatting with Prettier
terraform-validate Pre-submit Runs terraform validate on all root modules
helm-lint Pre-submit Lints Helm charts
check-rendered-files Pre-submit Verifies rendered deploy files are up to date
on-demand-e2e Pre-submit (manual) End-to-end: provisions ephemeral environment using PR rosa-regional-platform branch, runs tests, tears down. Trigger with /test on-demand-e2e on a PR
nightly-ephemeral Daily at 07:00 UTC End-to-end: provisions ephemeral environment using main rosa-regional-platform branch, runs tests, tears down
nightly-integration Daily at 07:00 UTC Runs e2e tests against a standing integration environment
ephemeral-resources-janitor Weekly (Sunday 12:00 UTC) Purges leaked AWS resources using aws-nuke

Build Image

The CI image is built from ci/Containerfile and includes all required tools (Terraform, Helm, AWS CLI, Python/uv, etc.).

Ephemeral Environment

The ci/ephemeral-provider/main.py script manages ephemeral environments for CI testing. It supports three modes — provision, teardown (--teardown), and resync (--resync) — designed to run as separate CI steps with tests in between.

  1. Creates a CI-owned git branch from the source repo/branch
  2. Bootstraps the pipeline-provisioner pointing at the CI branch
  3. Pushes rendered deploy files to trigger pipelines via GitOps
  4. Waits for RC/MC pipelines to provision infrastructure
  5. (Separate CI step) Runs the testing suite against the provisioned environment
  6. Tears down infrastructure via GitOps (delete: true in config.yaml)
  7. Destroys the pipeline-provisioner
  8. CI branch is retained for post-run troubleshooting (delete manually via git push ci --delete <branch>)

Running locally

The recommended way to run ephemeral environments locally is via Make targets. These handle container builds, Vault credential fetching, and state tracking automatically. Credentials are fetched from Vault via OIDC and passed as environment variables to the container — they never touch disk.

make ephemeral-provision   # Interactive remote/branch picker, provisions environment
make ephemeral-teardown    # Interactive picker or BUILD_ID=<id>, tears down environment
make ephemeral-resync      # Interactive picker or BUILD_ID=<id>, rebases CI branch onto latest source
make ephemeral-list        # List tracked environments with state

Prerequisites: fzf, vault, git, python3, uv, and podman or docker.

Triggering the E2E Job Manually

  1. Obtain an API token by visiting https://oauth-openshift.apps.ci.l2s4.p1.openshiftapps.com/oauth/token/request
  2. Log in with oc login
  3. Start the job:
# Trigger nightly-ephemeral
curl -X POST \
    -H "Authorization: Bearer $(oc whoami -t)" \
    'https://gangway-ci.apps.ci.l2s4.p1.openshiftapps.com/v1/executions/' \
    -d '{"job_name": "periodic-ci-openshift-online-rosa-regional-platform-main-nightly-ephemeral", "job_execution_type": "1"}'

# Trigger nightly-integration
curl -X POST \
    -H "Authorization: Bearer $(oc whoami -t)" \
    'https://gangway-ci.apps.ci.l2s4.p1.openshiftapps.com/v1/executions/' \
    -d '{"job_name": "periodic-ci-openshift-online-rosa-regional-platform-main-nightly-integration", "job_execution_type": "1"}'
  1. Copy the id from the response and check the execution to get the Prow URL:
curl -X GET \
    -H "Authorization: Bearer $(oc whoami -t)" \
    'https://gangway-ci.apps.ci.l2s4.p1.openshiftapps.com/v1/executions/<id>'

Open the job_url from the response to watch the job in Prow.

Accessing Live Job Logs

When a Prow job is running (e.g. on-demand-e2e), you can watch its logs in real time:

  1. Open the Prow job page (e.g. from the PR status check link or the job history -- see jobs table above).

  2. In the build log output, look for a line like:

    INFO[2026-03-10T11:41:49Z] Using namespace https://console.xxxxx.ci.openshift.org/k8s/cluster/projects/ci-op-XXXXXXXX

  3. Click the namespace link to open the OpenShift console for the CI cluster where the job pods are running. From there you can inspect pod logs, events, and resources in real time.

Note: Access to the namespace is restricted to the person who triggered the job (i.e. the PR author for pre-submit jobs). There is no configuration option to grant access to additional users.

CI Credentials

The e2e jobs use credentials mounted at /var/run/rosa-credentials/. Credentials are managed in Vault. Two credential secrets are used:

  • rosa-regional-platform-ephemeral-creds — grants access to the AWS accounts used to spin up an ephemeral environment. Used by nightly-ephemeral, on-demand-e2e, and ephemeral-resources-janitor.
  • rosa-regional-platform-integration-creds — grants access to AWS credentials for testing against the API gateway in the regional integration account. Used by nightly-integration.

Ephemeral Resources Janitor

The ephemeral tests create AWS resources across multiple accounts. Teardown relies on terraform destroy, which can fail and leak resources. The ephemeral-resources-janitor job is a weekly fallback that purges everything except resources we need to keep between tests using aws-nuke.

What is preserved

See ./ci/aws-nuke-config.yaml.

Running locally

# Dry-run (list only, no deletions)
./ci/janitor/purge-aws-account.sh

# Live run (actually delete resources)
./ci/janitor/purge-aws-account.sh --no-dry-run

The script uses whatever AWS credentials are active in your environment. The account must be in the allowlist in purge-aws-account.sh.