Skip to content

Emit first-party cluster-health metrics and ship a default alert-rule set in the Helm chart #357

Description

The operator currently exposes only stock controller-runtime metrics — the chart ships a ServiceMonitor, and the Cozystack packaging adds a VMServiceScrape, so the scrape surface exists — but no first-party health metrics and no alerting rules. That combination leaves the most damaging failure mode invisible: as #343 established, a cluster can enter a state the reconciler can never leave (e.g. scaleUp with zero live endpoints while status.clusterID is latched), and the loop retries forever while status.conditions keeps reporting Available=True. Anything watching conditions sees green; the only truthful signal is the reconcile loop itself.

Proposal, in two halves:

1. First-party metrics

Per EtcdCluster, labelled namespace/name (names are a strawman):

  • etcd_operator_cluster_members_desired / _members_ready / _members_learners
  • etcd_operator_cluster_quorum — 0/1 as currently probed by the operator, not as latched in status
  • etcd_operator_cluster_condition{condition=...} — terminal/degraded states (AllMembersLost from fix(controllers): report total member loss instead of spinning #343, BootstrapFailed, …)
  • etcd_operator_cluster_last_successful_reconcile_timestamp_seconds — the anti-lying heartbeat: consumers alert on its age instead of trusting conditions

2. A default alert-rule set shipped by the Helm chart

Values-gated (e.g. alerts.enabled), following the pattern cloudnative-pg and piraeus charts already use — the chart ships the rules, the cluster's evaluator just selects them:

  • Reconcile-error streak: rate(controller_runtime_reconcile_errors_total{controller=~"etcdcluster.*"}[10m]) > 0 sustained 1h → warning, 6h → critical. Catches every wedged-forever class, including the fix(controllers): report total member loss instead of spinning #343 one, on day one — no new instrumentation needed.
  • Terminal/degraded condition present → critical immediately.
  • members_ready < members_desired for 15m → critical.
  • quorum == 0 for 5m → critical.
  • time() - last_successful_reconcile > 1h per cluster → warning.

The upstream chart already ships prometheus-operator kinds (ServiceMonitor), so PrometheusRule fits there; victoria-metrics-operator conversion makes it consumable in Cozystack deployments, or the Cozystack package can mirror it as a VMRule next to its existing VMServiceScrape.

Separable follow-up: the operator holds the client TLS material for the etcd clusters it manages, so it is also the natural place to optionally render a per-cluster scrape plus the standard etcd data-plane rules (leader changes, fsync latency, db size vs quota). The five rules above don't depend on it.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions