You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The operator currently exposes only stock controller-runtime metrics — the chart ships a ServiceMonitor, and the Cozystack packaging adds a VMServiceScrape, so the scrape surface exists — but no first-party health metrics and no alerting rules. That combination leaves the most damaging failure mode invisible: as #343 established, a cluster can enter a state the reconciler can never leave (e.g. scaleUp with zero live endpoints while status.clusterID is latched), and the loop retries forever while status.conditions keeps reporting Available=True. Anything watching conditions sees green; the only truthful signal is the reconcile loop itself.
Proposal, in two halves:
1. First-party metrics
Per EtcdCluster, labelled namespace/name (names are a strawman):
etcd_operator_cluster_last_successful_reconcile_timestamp_seconds — the anti-lying heartbeat: consumers alert on its age instead of trusting conditions
2. A default alert-rule set shipped by the Helm chart
Values-gated (e.g. alerts.enabled), following the pattern cloudnative-pg and piraeus charts already use — the chart ships the rules, the cluster's evaluator just selects them:
Reconcile-error streak: rate(controller_runtime_reconcile_errors_total{controller=~"etcdcluster.*"}[10m]) > 0 sustained 1h → warning, 6h → critical. Catches every wedged-forever class, including the fix(controllers): report total member loss instead of spinning #343 one, on day one — no new instrumentation needed.
members_ready < members_desired for 15m → critical.
quorum == 0 for 5m → critical.
time() - last_successful_reconcile > 1h per cluster → warning.
The upstream chart already ships prometheus-operator kinds (ServiceMonitor), so PrometheusRule fits there; victoria-metrics-operator conversion makes it consumable in Cozystack deployments, or the Cozystack package can mirror it as a VMRule next to its existing VMServiceScrape.
Separable follow-up: the operator holds the client TLS material for the etcd clusters it manages, so it is also the natural place to optionally render a per-cluster scrape plus the standard etcd data-plane rules (leader changes, fsync latency, db size vs quota). The five rules above don't depend on it.
The operator currently exposes only stock controller-runtime metrics — the chart ships a ServiceMonitor, and the Cozystack packaging adds a VMServiceScrape, so the scrape surface exists — but no first-party health metrics and no alerting rules. That combination leaves the most damaging failure mode invisible: as #343 established, a cluster can enter a state the reconciler can never leave (e.g.
scaleUpwith zero live endpoints whilestatus.clusterIDis latched), and the loop retries forever whilestatus.conditionskeeps reportingAvailable=True. Anything watching conditions sees green; the only truthful signal is the reconcile loop itself.Proposal, in two halves:
1. First-party metrics
Per EtcdCluster, labelled
namespace/name(names are a strawman):etcd_operator_cluster_members_desired/_members_ready/_members_learnersetcd_operator_cluster_quorum— 0/1 as currently probed by the operator, not as latched in statusetcd_operator_cluster_condition{condition=...}— terminal/degraded states (AllMembersLostfrom fix(controllers): report total member loss instead of spinning #343,BootstrapFailed, …)etcd_operator_cluster_last_successful_reconcile_timestamp_seconds— the anti-lying heartbeat: consumers alert on its age instead of trusting conditions2. A default alert-rule set shipped by the Helm chart
Values-gated (e.g.
alerts.enabled), following the pattern cloudnative-pg and piraeus charts already use — the chart ships the rules, the cluster's evaluator just selects them:rate(controller_runtime_reconcile_errors_total{controller=~"etcdcluster.*"}[10m]) > 0sustained 1h → warning, 6h → critical. Catches every wedged-forever class, including the fix(controllers): report total member loss instead of spinning #343 one, on day one — no new instrumentation needed.members_ready < members_desiredfor 15m → critical.quorum == 0for 5m → critical.time() - last_successful_reconcile > 1hper cluster → warning.The upstream chart already ships prometheus-operator kinds (ServiceMonitor), so PrometheusRule fits there; victoria-metrics-operator conversion makes it consumable in Cozystack deployments, or the Cozystack package can mirror it as a VMRule next to its existing VMServiceScrape.
Separable follow-up: the operator holds the client TLS material for the etcd clusters it manages, so it is also the natural place to optionally render a per-cluster scrape plus the standard etcd data-plane rules (leader changes, fsync latency, db size vs quota). The five rules above don't depend on it.