OCPBUGS-67134: add grace period before reporting Available=False

[sg00dwin]

Summary

• During disruptive CI tests (node reboots, drains), console deployment replicas briefly drop to zero for ~10 seconds before self-recovering
• The operator immediately reports Available=False with Deployment_InsufficientReplicas, triggering OTA invariant test failures
• This fix adds a 15-second grace period that suppresses the false alarm when the deployment was recently available, while still reporting genuine outages promptly

What changed

• Track when the deployment was last seen available (lastDeploymentAvailableTime on the operator struct)
• Extract evaluateDeploymentAvailability() method that checks the grace window before reporting Available=False
• Grace period (15s) is kept short so genuine production outages are still reported promptly — long enough to absorb the observed ~10s blips but not so long that it masks real problems
• First startup with no prior healthy state reports immediately — no risk of masking real outages

Related

• OCPBUGS-67134
• Epic: CONSOLE-5185
• Follows the same operator-level tolerance pattern as #834 (route health retry), #1164 (degraded blips), #1169 (progressing/ObservedGeneration)

Test plan

• 7 new unit tests covering: available deployment, first-sync unavailable, grace period suppression, grace period expiry, recovery after blip, boundary condition, error message format
• All existing tests pass
• make test-unit clean (go test, gofmt, govet)
• After merge: monitor Sippy for ~5-7 days, then remove origin invariant test exception for console + Deployment_InsufficientReplicas

Co-Authored-By: Claude Opus 4.6

[openshift-ci-robot]

@sg00dwin: This pull request references Jira Issue OCPBUGS-67134, which is invalid:

• expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

• During disruptive CI tests (node reboots, drains), console deployment replicas briefly drop to zero for ~10 seconds before self-recovering
• The operator immediately reports Available=False with Deployment_InsufficientReplicas, triggering OTA invariant test failures
• This fix adds a 2-minute grace period that suppresses the false alarm when the deployment was recently available, while still reporting genuine outages

What changed

• Track when the deployment was last seen available (lastDeploymentAvailableTime on the operator struct)
• Extract evaluateDeploymentAvailability() method that checks the grace window before reporting Available=False
• Grace period (2 min) matches the existing Degraded inertia in library-go
• First startup with no prior healthy state reports immediately — no risk of masking real outages

Related

• OCPBUGS-67134
• Epic: CONSOLE-5185
• Follows the same operator-level tolerance pattern as #834 (route health retry), #1164 (degraded blips), #1169 (progressing/ObservedGeneration)

Test plan

• 7 new unit tests covering: available deployment, first-sync unavailable, grace period suppression, grace period expiry, recovery after blip, boundary condition, error message format
• All existing tests pass
• make test-unit clean (go test, gofmt, govet)
• After merge: monitor Sippy for ~5-7 days, then remove origin invariant test exception for console + Deployment_InsufficientReplicas

Co-Authored-By: Claude Opus 4.6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

Caution

Review failed

An error occurred during the review process. Please try again later.

Walkthrough

Adds timestamp tracking for console deployment availability and applies a 15-second grace period before reporting transient replica loss as unavailable. The availability logic is extracted into a helper, with tests covering initial, transient, expired, recovery, boundary, and error-message behavior.

Changes

Deployment availability handling

Layer / File(s)	Summary
Availability state and evaluation pkg/console/operator/operator.go, pkg/console/operator/sync_v400.go	Stores the last available timestamp, defines a 15-second grace period, routes sync evaluation through evaluateDeploymentAvailability, and reports InsufficientReplicas only after the grace period expires.
Availability evaluation tests pkg/console/operator/sync_v400_test.go	Tests availability initialization, transient replica loss, grace-period expiry, recovery, boundary timing, and replica-count error messages.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: jhadvig, therealjon

🚥 Pre-merge checks | ✅ 15

✅ Passed checks (15 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise, specific, and matches the main change: adding a grace period before reporting Available=False.
Description check	✅ Passed	It covers the root cause, solution, related issue, and test plan, though several template sections are omitted or folded into other headings.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The added subtest titles are static, descriptive strings with no dynamic values, timestamps, pod/node/namespace names, or other unstable identifiers.
Test Structure And Quality	✅ Passed	New tests are plain table-driven unit tests, cover one scenario per subtest, use no cluster resources/waits, and follow existing repo test style.
Microshift Test Compatibility	✅ Passed	The new test is a plain Go unit test (testing.T), not a Ginkgo e2e test, and it only uses Deployments/core APIs available on MicroShift.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	No new Ginkgo/e2e test was added; the new coverage is a plain unit test on deployment status and has no multi-node/SNO assumptions.
Topology-Aware Scheduling Compatibility	✅ Passed	Touched code only adds deployment availability grace logic and tests; no pod specs, replicas, affinity, node selectors, or topology labels were introduced.
Ote Binary Stdout Contract	✅ Passed	Touched code only adds availability logic and tests; no stdout writes or process-level setup changes appear in main/init/TestMain/BeforeSuite paths.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	PASS: The added tests are standard unit tests; no Ginkgo e2e blocks, hardcoded IPv4s, or external connectivity were found.
No-Weak-Crypto	✅ Passed	No MD5/SHA1/DES/RC4/3DES/Blowfish/ECB, custom crypto, or secret/token comparisons were added in the touched files.
Container-Privileges	✅ Passed	PR only changes Go logic/tests; no privileged/hostNetwork/hostPID/hostIPC/SYS_ADMIN/allowPrivilegeEscalation settings found in changed files or repo scan.
No-Sensitive-Data-In-Logs	✅ Passed	No new logging of secrets, tokens, PII, hostnames, or customer data was added; the only new log reports replica timing and grace-period duration.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

Warning

Tools execution failed with the following error:

Failed to run tools: 13 INTERNAL: Received RST_STREAM with code 2 (Internal server error)

---

_{Comment @coderabbitai help to get the list of available commands.}

[sg00dwin]

/retest

[sg00dwin]

/jira refresh

[openshift-ci-robot]

@sg00dwin: This pull request references Jira Issue OCPBUGS-67134, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira (yapei@redhat.com), skipping review request.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[sg00dwin]

/retest-required

[sg00dwin]

/test e2e-aws-console

[sg00dwin]

/test e2e-aws-console

[jhadvig]

@sg00dwin good investigation on this one, the root cause analysis is solid and the tests are well written 👍

I've been thinking about this more though, and I'm not sure suppressing Available=False for 2 minutes is the right fix here. Available is deliberately given no inertia in library-go - it's meant to be a real-time signal of "can users use this right now?". Reporting Available=True when replicas are actually at zero is lying about cluster state, and CVO/monitoring rely on that signal during upgrades and alerting.

The real question is: why are all replicas going down during a single node reboot? If they're colocated on the same node, that's the root cause we should fix.

A few alternatives worth exploring:

1. Pod topology spread constraints / anti-affinity - ensure console replicas land on different nodes so a single node disruption can't take all of them out. This fixes the actual problem rather than masking it.

2. PodDisruptionBudget - if there isn't one already, a PDB with minAvailable: 1 would prevent voluntary drains from evicting all replicas simultaneously.

3. Fix it at the test/invariant layer - if conformance-serial tests do disruptive node operations that are expected to cause brief disruptions, the invariant tests should tolerate brief Available=False during those windows rather than the operator hiding it. Could be worth discussing with the TRT team.

4. Propose Available inertia in library-go - if this is a real gap that other operators also hit, the proper fix is adding WithAvailableInertia to StatusSyncer so it's done consistently for everyone, not ad-hoc per operator.

Can you dig into whether we have topology constraints on the console deployment? That feels like the most impactful fix here.

[sg00dwin]

/test e2e-aws-console

[sg00dwin]

@sg00dwin good investigation on this one, the root cause analysis is solid and the tests are well written 👍

I've been thinking about this more though, and I'm not sure suppressing Available=False for 2 minutes is the right fix here. Available is deliberately given no inertia in library-go - it's meant to be a real-time signal of "can users use this right now?". Reporting Available=True when replicas are actually at zero is lying about cluster state, and CVO/monitoring rely on that signal during upgrades and alerting.

The real question is: why are all replicas going down during a single node reboot? If they're colocated on the same node, that's the root cause we should fix.

A few alternatives worth exploring:

1. Pod topology spread constraints / anti-affinity - ensure console replicas land on different nodes so a single node disruption can't take all of them out. This fixes the actual problem rather than masking it.
2. PodDisruptionBudget - if there isn't one already, a PDB with minAvailable: 1 would prevent voluntary drains from evicting all replicas simultaneously.
3. Fix it at the test/invariant layer - if conformance-serial tests do disruptive node operations that are expected to cause brief disruptions, the invariant tests should tolerate brief Available=False during those windows rather than the operator hiding it. Could be worth discussing with the TRT team.
4. Propose Available inertia in library-go - if this is a real gap that other operators also hit, the proper fix is adding WithAvailableInertia to StatusSyncer so it's done consistently for everyone, not ad-hoc per operator.

Can you dig into whether we have topology constraints on the console deployment? That feels like the most impactful fix here.

@jhadvig Thanks for the thorough review!

On suggestions 1 and 2 the console deployment already has both:

• Hard anti-affinity (RequiredDuringSchedulingIgnoredDuringExecution on kubernetes.io/hostname) so pods land on different nodes in HA mode
• PDB with maxUnavailable: 1 preventing voluntary drains from taking both pods down

The blip still happens because the conformance-serial tests do involuntary node reboots, which bypass the PDB. When the test hits the right nodes, both pods go offline for ~10 seconds regardless.

On option 3 - there's already an origin exception demoting this to a flake, but OTA-362 is moving to remove those exceptions rather than add smarter ones. So the test layer is heading in the opposite direction.

The operator-level grace period was modeled after the sibling fixes (OCPBUGS-24041, OCPBUGS-38676, OCPBUGS-64688), but your point about Available being intentionally real-time is a fair one. Would you prefer we pursue the library-go path instead — proposing inertia for Available on StatusSyncer so it's handled consistently across operators? Happy to go either direction.

[sg00dwin]

/test e2e-aws-console

[sg00dwin]

/test e2e-aws-console

[sg00dwin]

4. Propose Available inertia in library-go - if this is a real gap that other operators also hit, the proper fix is adding WithAvailableInertia to StatusSyncer so it's done consistently for everyone, not ad-hoc per operator.

Suggesting can we go ahead and merge this pr fix ... and I will open a follow-on issue to track the library-go WithAvailableInertia approach so all operators benefit long-term? Also, need to know what component to assign that bug to.

[openshift-ci-robot]

@sg00dwin: This pull request references Jira Issue OCPBUGS-67134, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira (yapei@redhat.com), skipping review request.

Details

In response to this:

Summary

• During disruptive CI tests (node reboots, drains), console deployment replicas briefly drop to zero for ~10 seconds before self-recovering
• The operator immediately reports Available=False with Deployment_InsufficientReplicas, triggering OTA invariant test failures
• This fix adds a 2-minute grace period that suppresses the false alarm when the deployment was recently available, while still reporting genuine outages

What changed

• Track when the deployment was last seen available (lastDeploymentAvailableTime on the operator struct)
• Extract evaluateDeploymentAvailability() method that checks the grace window before reporting Available=False
• Grace period (2 min) matches the existing Degraded inertia in library-go
• First startup with no prior healthy state reports immediately — no risk of masking real outages

Related

• OCPBUGS-67134
• Epic: CONSOLE-5185
• Follows the same operator-level tolerance pattern as #834 (route health retry), #1164 (degraded blips), #1169 (progressing/ObservedGeneration)

Test plan

• 7 new unit tests covering: available deployment, first-sync unavailable, grace period suppression, grace period expiry, recovery after blip, boundary condition, error message format
• All existing tests pass
• make test-unit clean (go test, gofmt, govet)
• After merge: monitor Sippy for ~5-7 days, then remove origin invariant test exception for console + Deployment_InsufficientReplicas

Co-Authored-By: Claude Opus 4.6

Summary by CodeRabbit

• Bug Fixes

• Improved console deployment availability detection by adding a short grace period (15 seconds) after the deployment was last available.

• Transient replica drops no longer trigger brief “unavailable/insufficient replicas” alerts; the status only degrades after the grace window expires.

• When replicas recover after a suppressed blip, availability status updates and resumes normal reporting.

• Tests

• Added coverage to verify grace-period behavior and expected “InsufficientReplicas” messaging.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[sg00dwin]

/test e2e-aws-console

[sg00dwin]

/test e2e-aws-console

[jhadvig]

/label acknowledge-critical-fixes-only
/lgtm
/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jhadvig, sg00dwin

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [jhadvig]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sg00dwin]

/test e2e-aws-console

[sg00dwin]

Verified on a CI cluster with the PR changes deployed — no regressions observed.

Cluster: OCP on AWS (ci-ln-2xqz9vt)

Steps performed:

1. Logged into the cluster and confirmed console operator healthy (Available=True, 2/2 replicas)
2. Reviewed the 3 changed files — new evaluateDeploymentAvailability() method adds a 15s grace period before reporting Available=False with InsufficientReplicas
3. Ran unit tests — all 19 pass including 7 new tests covering grace period suppression, expiry, first-sync, recovery, boundary, and error format
4. Verified consumer tests (deployment.IsAvailable, telemetry) — no regressions
5. Full make test-unit clean (gofmt, govet, all packages)
6. Checked cluster logs — first-sync correctly reported InsufficientReplicas immediately (no prior healthy state, no grace period applied)

/verified by @sg00dwin and Claude

[openshift-ci-robot]

@sg00dwin: This PR has been marked as verified by @sg00dwin and Claude.

Details

In response to this:

Verified on a CI cluster with the PR changes deployed — no regressions observed.

Cluster: OCP on AWS (ci-ln-2xqz9vt)

Steps performed:

1. Logged into the cluster and confirmed console operator healthy (Available=True, 2/2 replicas)
2. Reviewed the 3 changed files — new evaluateDeploymentAvailability() method adds a 15s grace period before reporting Available=False with InsufficientReplicas
3. Ran unit tests — all 19 pass including 7 new tests covering grace period suppression, expiry, first-sync, recovery, boundary, and error format
4. Verified consumer tests (deployment.IsAvailable, telemetry) — no regressions
5. Full make test-unit clean (gofmt, govet, all packages)
6. Checked cluster logs — first-sync correctly reported InsufficientReplicas immediately (no prior healthy state, no grace period applied)

/verified by @sg00dwin and Claude

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

@sg00dwin: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci-robot]

@sg00dwin: Jira Issue Verification Checks: Jira Issue OCPBUGS-67134
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-67134 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

Summary

• During disruptive CI tests (node reboots, drains), console deployment replicas briefly drop to zero for ~10 seconds before self-recovering
• The operator immediately reports Available=False with Deployment_InsufficientReplicas, triggering OTA invariant test failures
• This fix adds a 15-second grace period that suppresses the false alarm when the deployment was recently available, while still reporting genuine outages promptly

What changed

• Track when the deployment was last seen available (lastDeploymentAvailableTime on the operator struct)
• Extract evaluateDeploymentAvailability() method that checks the grace window before reporting Available=False
• Grace period (15s) is kept short so genuine production outages are still reported promptly — long enough to absorb the observed ~10s blips but not so long that it masks real problems
• First startup with no prior healthy state reports immediately — no risk of masking real outages

Related

• OCPBUGS-67134
• Epic: CONSOLE-5185
• Follows the same operator-level tolerance pattern as #834 (route health retry), #1164 (degraded blips), #1169 (progressing/ObservedGeneration)

Test plan

• 7 new unit tests covering: available deployment, first-sync unavailable, grace period suppression, grace period expiry, recovery after blip, boundary condition, error message format
• All existing tests pass
• make test-unit clean (go test, gofmt, govet)
• After merge: monitor Sippy for ~5-7 days, then remove origin invariant test exception for console + Deployment_InsufficientReplicas

Co-Authored-By: Claude Opus 4.6

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-merge-robot]

Fix included in release 5.0.0-0.nightly-2026-07-15-061255