Skip to content

feat: add rhdh.redhat.com/idle annotation for idling/waking workloads - #3318

Open
rm3l wants to merge 5 commits into
redhat-developer:mainfrom
rm3l:RHIDP-15995--add-rhdh-redhat-com-idle-annotation-support-to-the-operator-controller
Open

feat: add rhdh.redhat.com/idle annotation for idling/waking workloads#3318
rm3l wants to merge 5 commits into
redhat-developer:mainfrom
rm3l:RHIDP-15995--add-rhdh-redhat-com-idle-annotation-support-to-the-operator-controller

Conversation

@rm3l

@rm3l rm3l commented Aug 4, 2026

Copy link
Copy Markdown
Member

Description

Adds support for the rhdh.redhat.com/idle annotation on the Backstage CR. When set to "true", the operator overrides replicas to 0 on the Backstage Deployment and (if enabled) the local DB StatefulSet. Removing the annotation triggers a wake-up, restoring replicas to the user's patched value or defaulting to 1.

This enables external controllers (e.g. Dev Sandbox idler) to scale RHDH instances to zero without conflicting with the operator's Server-Side Apply field ownership of the replicas field.

State machine:

  • Normal: replicas not managed by idle logic (HPA-friendly)
  • Idle (annotation="true"): replicas forced to 0, status condition reason set to Idled
  • Wake (annotation removed, was previously Idled): replicas restored to patch value or 1, then released on next reconcile so HPA can take over

External DB compatibility: When spec.database.enableLocalDb=false, only the Backstage Deployment is affected. The external DB is never touched.

Which issue(s)

Fixes RHIDP-15995

Part of: RHIDP-15996, RHDHPLAN-1040

PR acceptance criteria

  • Tests
  • Documentation

How to test changes / Special notes to the reviewer

# Tests should pass
make test integration-test

# Manual test on a cluster
oc annotate backstage developer-hub rhdh.redhat.com/idle=true
# Verify pods scale to 0, status shows Idled
oc annotate backstage developer-hub rhdh.redhat.com/idle-
# Verify pods scale back to 1, status shows Deployed

Building Container Images for Testing

Need to test container images from this PR?

For Maintainers: To trigger a test image build, review the code and comment /build-images.
This always builds the HEAD of the PR branch.

For Contributors: Ask a maintainer to run /build-images.

Images will be built and pushed to Quay with links posted in comments.

…nstances

When the `rhdh.redhat.com/idle: "true"` annotation is set on a Backstage CR,
the operator overrides replicas to 0 on the Backstage Deployment and (if
enabled) the local DB StatefulSet. Removing the annotation triggers a wake-up,
restoring replicas to the user's patched value or defaulting to 1.

This enables external controllers (e.g. Dev Sandbox idler) to scale RHDH
instances to zero without conflicting with the operator's Server-Side Apply
field ownership of the replicas field.

State machine:
- Normal: replicas not managed by idle logic (HPA-friendly)
- Idle (annotation="true"): replicas forced to 0, status reason set to "Idled"
- Wake (annotation removed, was Idled): replicas restored, then released

Ref: RHIDP-15995

Assisted-by: Claude
@rm3l rm3l changed the title feat: add rhdh.redhat.com/idle annotation support for idling/waking instances feat: add rhdh.redhat.com/idle annotation for idling/waking workloads Aug 4, 2026
@codecov

codecov Bot commented Aug 4, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 54.54545% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.54%. Comparing base (845a0e9) to head (a02dddc).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
internal/controller/backstage_controller.go 0.00% 3 Missing ⚠️
pkg/model/statefulset_obj.go 0.00% 2 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3318      +/-   ##
==========================================
+ Coverage   63.49%   63.54%   +0.04%     
==========================================
  Files          38       38              
  Lines        2356     2367      +11     
==========================================
+ Hits         1496     1504       +8     
- Misses        711      714       +3     
  Partials      149      149              
Flag Coverage Δ
nightly ?
unittests 63.54% <54.54%> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
pkg/model/db-statefulset.go 96.29% <100.00%> (+0.14%) ⬆️
pkg/model/deployable.go 75.00% <ø> (ø)
pkg/model/deployment.go 81.67% <100.00%> (+0.14%) ⬆️
pkg/model/deployment_obj.go 83.72% <100.00%> (+5.67%) ⬆️
pkg/model/runtime.go 78.88% <ø> (ø)
pkg/model/statefulset_obj.go 14.28% <0.00%> (-0.72%) ⬇️
internal/controller/backstage_controller.go 0.00% <0.00%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

rm3l added 2 commits August 4, 2026 09:02
Documents the rhdh.redhat.com/idle annotation: how to idle and wake
instances, behavior with local vs external DB, HPA compatibility,
and status condition transitions.

Ref: RHIDP-15995

Assisted-by: Claude
The CI runs with USE_EXISTING_CONTROLLER=true, where ReconcileAny is a
no-op and reconciliation happens asynchronously via the background
controller. Synchronous assertions after ReconcileAny fail because the
controller hasn't processed the annotation change yet.

Wrap all post-idle and post-wake assertions in Eventually blocks to poll
until the controller reconciles.

Ref: RHIDP-15995

Assisted-by: Claude

@gazarenkov gazarenkov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why that complex?
IMO the logic is just:

  • check the annotation (in corresponding addToModel or updateAndValidate methods)
  • if idled set replica=0 in deployment, statefulset
  • otherwise keep unchanged
    Did I miss something?

@rm3l

rm3l commented Aug 4, 2026

Copy link
Copy Markdown
Member Author

Why that complex? IMO the logic is just:

* check the annotation (in corresponding addToModel or updateAndValidate methods)

* if idled set replica=0 in deployment, statefulset

* otherwise keep unchanged
  Did I miss something?

Yeah, good point. This is just the initial draft Claude implementation for this. Still refining it :). Thanks.

Move the idle annotation check (replicas=0) into the updateAndValidate
methods of BackstageDeployment and DbStatefulSet, removing the separate
idle/wake block from the controller. Wake logic is unnecessary because
the model is rebuilt from scratch on every reconcile — when the annotation
is removed, replicas naturally get their default or patched value.

Assisted-by: Claude
@rm3l
rm3l force-pushed the RHIDP-15995--add-rhdh-redhat-com-idle-annotation-support-to-the-operator-controller branch from e4120f2 to 18a27ea Compare August 4, 2026 08:17
@rm3l
rm3l marked this pull request as ready for review August 4, 2026 08:52
@rm3l
rm3l requested a review from a team as a code owner August 4, 2026 08:52
@rhdh-qodo-merge

Copy link
Copy Markdown

PR Summary by Qodo

Add rhdh.redhat.com/idle annotation to scale Backstage (and local DB) to zero

✨ Enhancement 🧪 Tests 📝 Documentation 🕐 40+ Minutes

Grey Divider

AI Description

• Add rhdh.redhat.com/idle: "true" support to idle instances by forcing replicas=0.
• Report idled state via a new Idled status condition reason.
• Add unit/integration coverage and document idling/waking behavior and DB/HPA considerations.
Diagram

graph TD
  CR["Backstage CR (idle anno)"] --> CTRL["BackstageReconciler"] --> MODEL["model.InitObjects()"] --> APPLY["applyObjects (SSA)"] --> DEP["Backstage workload"]
  APPLY --> DB["Local DB SS"]
  CTRL --> STATUS["Status Reason=Idled"]

  subgraph Legend
    direction LR
    _api{{"API/CR"}} ~~~ _svc(["Controller"]) ~~~ _mod["Model code"] ~~~ _k8s[("K8s workload")]
  end
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Promote idling to a spec field (e.g. spec.idle) instead of an annotation
  • ➕ Schema validation and better discoverability via CRD/OpenAPI and kubectl explain
  • ➕ Clearer contract for users and tools that generate CRs
  • ➖ Harder for external idlers to toggle without modifying spec (potentially more field-manager conflicts)
  • ➖ Requires API evolution/versioning and more conversion/compat work
2. Manage replicas via scale subresource / dedicated field manager strategy
  • ➕ Cleaner ownership separation for replicas without embedding behavior into object generation
  • ➕ Could reduce SSA contention if ownership boundaries are carefully designed
  • ➖ More complex reconcile/patch strategy and more K8s edge cases (scale subresource, SSA ownership)
  • ➖ May still conflict with SSA depending on how patches are applied

Recommendation: The annotation-driven approach is well aligned with the goal of letting external controllers scale to zero without fighting the operator’s SSA ownership. Centralizing the behavior in model generation (force replicas only when idled) keeps external DB untouched and limits blast radius. If this becomes a broadly user-facing feature, consider a future spec field for stronger validation/discoverability.

Files changed (12) +297 / -0

Enhancement (6) +18 / -0
current-types.goExpose Idled condition reason in current API aliases +1/-0

Expose Idled condition reason in current API aliases

• Adds 'BackstageConditionReasonIdled' to the current API constants so controller code can use the reason consistently across API versions.

api/current-types.go

backstage_types.goAdd Idled to Backstage condition reasons +1/-0

Add Idled to Backstage condition reasons

• Introduces 'BackstageConditionReasonIdled' in v1alpha5, enabling status to explicitly represent an idled instance state.

api/v1alpha5/backstage_types.go

backstage_controller.goSet Idled status when idle annotation is present +5/-0

Set Idled status when idle annotation is present

• After applying runtime objects, detects 'rhdh.redhat.com/idle="true"' on the Backstage CR, sets the deployed condition to False with reason 'Idled', and returns early instead of reporting normal deployment status.

internal/controller/backstage_controller.go

db-statefulset.goForce local DB replicas to 0 when idled +6/-0

Force local DB replicas to 0 when idled

• When local DB is enabled and the Backstage CR has the idle annotation set to true, sets the DB StatefulSet '.spec.replicas' to 0 during model update/validation.

pkg/model/db-statefulset.go

deployment.goForce Backstage workload replicas to 0 when idled +4/-0

Force Backstage workload replicas to 0 when idled

• When the Backstage CR has the idle annotation set to true, sets the generated Backstage Deployment/StatefulSet replicas to 0 via the Deployable interface.

pkg/model/deployment.go

runtime.goDefine IdleAnnotation constant in model runtime package +1/-0

Define IdleAnnotation constant in model runtime package

• Introduces the 'IdleAnnotation' constant ('rhdh.redhat.com/idle') for consistent reference across controller/model/tests.

pkg/model/runtime.go

Refactor (3) +10 / -0
deployable.goExtend Deployable abstraction with SetReplicas +2/-0

Extend Deployable abstraction with SetReplicas

• Adds a 'SetReplicas(*int32)' method to the Deployable interface so replica mutations can be applied uniformly across Deployment and StatefulSet implementations.

pkg/model/deployable.go

deployment_obj.goImplement SetReplicas for Deployment-backed Deployable +4/-0

Implement SetReplicas for Deployment-backed Deployable

• Implements 'SetReplicas' on 'DeploymentObj' to support replica overrides through the Deployable interface.

pkg/model/deployment_obj.go

statefulset_obj.goImplement SetReplicas for StatefulSet-backed Deployable +4/-0

Implement SetReplicas for StatefulSet-backed Deployable

• Implements 'SetReplicas' on 'StatefulSetObj', completing the Deployable interface extension across supported workload kinds.

pkg/model/statefulset_obj.go

Tests (2) +217 / -0
idle_test.goAdd integration coverage for idling/waking behavior +179/-0

Add integration coverage for idling/waking behavior

• Adds Ginkgo tests verifying that setting the idle annotation scales the Backstage workload (and local DB StatefulSet) to 0 and sets status reason to Idled, and that removing the annotation restores replicas and clears the Idled reason. Uses 'Eventually' blocks to tolerate async reconciliation in CI modes.

integration_tests/idle_test.go

deployment_test.goAdd unit tests for idle replica overrides (local/external DB) +38/-0

Add unit tests for idle replica overrides (local/external DB)

• Adds unit tests ensuring idle annotation sets Backstage replicas to 0 and, when local DB is enabled, also sets DB StatefulSet replicas to 0; verifies external DB configuration only impacts the Backstage workload.

pkg/model/deployment_test.go

Documentation (1) +52 / -0
configuration.mdDocument instance idling via rhdh.redhat.com/idle +52/-0

Document instance idling via rhdh.redhat.com/idle

• Adds an “Instance Idling” section describing how the annotation forces replicas to 0, how waking works, and how local vs external DB is handled, including example commands and expected status condition output.

docs/configuration.md

@rhdh-qodo-merge

rhdh-qodo-merge Bot commented Aug 4, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 🔗 Cross-repo conflicts (1) 📜 Skill insights (0)

Context used
⚠️ Tickets: not configured — ticket URL found in PR but could not be fetched — check ticket provider credentials
✅ Compliance rules (platform): 18 rules
✅ Cross-repo context
  Explored: repo: redhat-developer/rhdh (sha: 3875f708)
  Not relevant to this PR: redhat-developer/rhdh-plugins

Grey Divider


Remediation recommended

1. Idling docs omit namespace rule ✓ Resolved 📘 Rule violation ✧ Quality
Description
The new Instance Idling documentation describes changing replicas on the Backstage
Deployment/StatefulSet and local DB StatefulSet but does not explicitly state the namespace where
these resources are managed, which can cause user confusion in multi-namespace setups.
Code

docs/configuration.md[R923-926]

+When the annotation `rhdh.redhat.com/idle` is set to `"true"` on the Backstage CR, the Operator overrides replicas to 0 on both the Backstage Deployment (or StatefulSet) and the local DB StatefulSet (if enabled). The status condition reason is set to `Idled`.
+
+When the annotation is removed (or set to any value other than `"true"`), the Operator restores replicas:
+- If the user specified replicas via `spec.deployment.patch`, that value is preserved.
Relevance

●●● Strong

Team has accepted clarifying namespace/CR location in docs to avoid ambiguity.

PR-#1551

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
PR Compliance ID 5 requires docs to explicitly document the namespace where namespaced resources are
created/managed when relevant. The new Instance Idling section describes mutating replicas on
Deployments/StatefulSets but does not specify they are managed in the same namespace as the
Backstage CR.

Rule 5: Document resource creation namespace when behaviorally relevant
docs/configuration.md[919-932]
pkg/model/runtime.go[212-216]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The Instance Idling docs do not state where the affected workload resources live (namespace behavior), even though the operator manages namespaced resources.

## Issue Context
Compliance requires user-facing docs to explicitly document namespace behavior when it is behaviorally relevant.

## Fix Focus Areas
- docs/configuration.md[919-933]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. Idle patch test missing ✓ Resolved 🐞 Bug ⚙ Maintainability
Description
The integration test "wakes with user-specified replicas from deployment patch" never sets
spec.deployment.patch, so it only validates the default wake behavior and can’t catch regressions
in restoring patched replica counts.
Code

integration_tests/idle_test.go[R144-147]

+		By("removing idle annotation and adding replicas via deployment patch")
+		Expect(k8sClient.Get(ctx, types.NamespacedName{Name: backstageName, Namespace: ns}, bs)).To(Succeed())
+		delete(bs.Annotations, model.IdleAnnotation)
+		Expect(k8sClient.Update(ctx, bs)).To(Succeed())
Relevance

●●● Strong

They commonly accept tightening integration tests to cover intended behaviors/regressions.

PR-#1406

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The test constructs the Backstage CR with an empty spec and later only removes the idle annotation;
it never assigns spec.deployment.patch. The API and model code show that the patch field is the
mechanism intended to drive user-specified replica restoration.

integration_tests/idle_test.go[114-160]
api/v1alpha5/backstage_types.go[57-68]
pkg/model/deployment.go[224-270]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The integration test claims to verify waking with a user-specified replica count via `spec.deployment.patch`, but it never sets `Backstage.spec.deployment.patch`. As written, it only proves wake restores to the default (1), leaving the patch-restore behavior untested.

## Issue Context
- `BackstageDeployment.Patch` exists in the API and is merged into the deployable object during model initialization.
- To validate the behavior described in the PR, the test should set a non-default `replicas` value via `spec.deployment.patch`, idle the instance, then wake it and assert replicas returns to that non-default value.

## Fix Focus Areas
- integration_tests/idle_test.go[114-160]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. Idled blocks CI waits 🔗 Cross-repo conflict ☼ Reliability
Description
This PR sets the Backstage Deployed condition to Status=False with Reason=Idled whenever
rhdh.redhat.com/idle=true is present. The redhat-developer/rhdh CI helper waits only for
Deployed.Status==True, so runs on clusters where an external idler sets this annotation can time
out instead of failing fast or waking the instance.
Code

internal/controller/backstage_controller.go[R117-120]

+	if backstage.GetAnnotations()[model.IdleAnnotation] == "true" {
+		setStatusCondition(&backstage, api.BackstageConditionTypeDeployed, metav1.ConditionFalse, api.BackstageConditionReasonIdled, "Instance is idled")
+		return ctrl.Result{}, nil
+	}
Relevance

●● Moderate

Potential CI/runtime behavior impact; no clear precedent on handling Idled vs Deployed readiness
semantics.

PR-#1949

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
In the operator PR, idled instances are explicitly marked Deployed=False with Reason=Idled. In
the rhdh repo’s CI library, k8s_wait::backstage_resource() polls only for `Deployed.Status ==
True` and otherwise waits until timeout, so an idled instance will never satisfy the readiness
check.

internal/controller/backstage_controller.go[117-120]
External repo: redhat-developer/rhdh, .ci/pipelines/lib/k8s-wait.sh [245-280]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The operator now reports an idled instance as `Deployed=False` / `Reason=Idled`. RHDH CI’s `k8s_wait::backstage_resource()` currently treats **only** `Deployed.Status=True` as success, so if an external controller sets `rhdh.redhat.com/idle=true` (which this PR explicitly enables), CI can burn the full timeout and produce confusing failures.

## Issue Context
This is a cross-repo contract/assumption mismatch between:
- **Provider**: rhdh-operator setting `Deployed=False` when idled.
- **Consumer**: rhdh CI scripts waiting for `Deployed=True` without checking `Reason`.

## Fix Focus Areas
- internal/controller/backstage_controller.go[117-120]
- /cross_repos/rhdh/.ci/pipelines/lib/k8s-wait.sh[245-280]

## Suggested fix approaches
Choose one (or combine):
1. **Update rhdh CI wait helper** to also fetch the `Deployed` condition `reason`; if `reason == "Idled"`, fail fast with a clear message (and optionally auto-wake by removing the annotation behind an env flag).
2. **Update rhdh CI install flows** to ensure the Backstage CR has no `rhdh.redhat.com/idle` annotation before waiting for readiness.

Keep the operator semantics (idled means not deployed/running) intact; make the CI logic explicit about handling this new, valid state.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Informational

4. Backstage YAML lacks CRD reference ✓ Resolved 📘 Rule violation ✧ Quality
Description
The new declarative YAML example uses the Backstage custom resource but does not include (or link
to) the required Backstage CRD, so the example set is not self-applicable as written.
Code

docs/configuration.md[R942-945]

+apiVersion: rhdh.redhat.com/v1alpha5
+kind: Backstage
+metadata:
+  name: my-backstage
Relevance

● Weak

Close precedent rejected adding CRD install/reference context to Backstage CR YAML example docs.

PR-#2717

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
PR Compliance ID 1 requires example manifests to include all dependent resources (including CRDs) or
provide a clear reference to the CRD manifest. The new snippet defines a Backstage custom resource
but provides no CRD definition or reference alongside it.

Rule 1: Include all dependent Kubernetes resources in example manifests
docs/configuration.md[939-949]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new YAML example for `kind: Backstage` is not self-contained because it relies on a CRD that is not included or referenced in the same doc/example block.

## Issue Context
Compliance requires example manifest bundles/snippets to include dependent resources (including CRDs) or clearly reference where they come from, so a reader can conceptually `kubectl apply -f` the example set without missing-kind errors.

## Fix Focus Areas
- docs/configuration.md[939-949]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

To customize comments, go to the Qodo configuration screen, or learn more in the docs.

Qodo Logo

@rhdh-qodo-merge rhdh-qodo-merge Bot added documentation Improvements or additions to documentation enhancement New feature or request Tests Bug fix labels Aug 4, 2026
@rm3l

rm3l commented Aug 4, 2026

Copy link
Copy Markdown
Member Author

/agentic_review

@redhat-developer redhat-developer deleted a comment from github-actions Bot Aug 4, 2026
@rhdh-qodo-merge

Copy link
Copy Markdown

Code review by qodo was updated up to the latest commit 78b8f8a

- Document namespace scope, CRD prerequisite, and CI/monitoring
  contract for the Idled status condition
- Fix integration test to actually set replicas via spec.deployment.patch
  and assert wake restores to the patched value (3), not just the default

Assisted-by: Claude
@rm3l
rm3l force-pushed the RHIDP-15995--add-rhdh-redhat-com-idle-annotation-support-to-the-operator-controller branch from 78b8f8a to a02dddc Compare August 4, 2026 09:29
@sonarqubecloud

sonarqubecloud Bot commented Aug 4, 2026

Copy link
Copy Markdown

@redhat-developer redhat-developer deleted a comment from github-actions Bot Aug 4, 2026
@rm3l

rm3l commented Aug 4, 2026

Copy link
Copy Markdown
Member Author

/build-images

@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

PR images built successfully!

Images are available for testing:

  1. Operator: quay.io/rhdh-community/operator:2.0.0-pr-3318-a02dddc
  2. Bundle: quay.io/rhdh-community/operator-bundle:2.0.0-pr-3318-a02dddc
  3. Catalog: quay.io/rhdh-community/operator-catalog:2.0.0-pr-3318-a02dddc

Also available with PR number tag:

  • quay.io/rhdh-community/operator:2.0.0-pr-3318
  • quay.io/rhdh-community/operator-bundle:2.0.0-pr-3318
  • quay.io/rhdh-community/operator-catalog:2.0.0-pr-3318

Triggered by @rm3l

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Bug fix documentation Improvements or additions to documentation enhancement New feature or request Tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants