diff --git a/.coverage-baseline b/.coverage-baseline index 2ce0a0b6..3906a389 100644 --- a/.coverage-baseline +++ b/.coverage-baseline @@ -1 +1 @@ -78.4 +78.5 diff --git a/.docs-lint-scope b/.docs-lint-scope index 48e2793a..3834a34d 100644 --- a/.docs-lint-scope +++ b/.docs-lint-scope @@ -25,3 +25,6 @@ README.md docs/architecture.md docs/configuration.md +docs/design/attribution-branch-findings.md +docs/design/attribution-publish-and-join.md +docs/design/attribution-metrics-proposal.md diff --git a/charts/gitops-reverser/README.md b/charts/gitops-reverser/README.md index 0903d7c9..8b46641d 100644 --- a/charts/gitops-reverser/README.md +++ b/charts/gitops-reverser/README.md @@ -186,7 +186,7 @@ nodeSelector: | `quickstart.gitProvider.secretRef.name` | Existing Secret name used by the starter `GitProvider` | `git-creds` | | `quickstart.gitTarget.path` | Repository path used by the starter `GitTarget`; set `.` only to deliberately target the repo root | `live-cluster` | | `quickstart.watchRule.rules` | Rules used by the starter `WatchRule` | `configmaps create/update/delete` | -| `queue.redis.addr` | Redis/Valkey endpoint (`host:port`). Optional but advised: empty runs `configured-author` with cold-replay on restart. Set it for warm-restart cursors and for the admission webhook to actually record CommitRequest authors (admission runs as a no-op without it); **required** only when `attribution.enabled=true` | `""` | +| `queue.redis.addr` | Redis/Valkey endpoint (`host:port`). Optional but advised: empty means cold-replay on restart. Set it for warm-restart cursors and for the admission webhook to actually record CommitRequest authors (admission runs as a no-op without it); **required** when `attribution.enabled=true` unless `attribution.transport=memory` | `""` | | `queue.redis.auth.existingSecret` | Name of a pre-created Secret holding the Redis password (only used when `queue.redis.addr` is set) | `""` | | `queue.redis.auth.existingSecretKey` | Key within the Secret that holds the password | `password` | | `queue.redis.auth.username` | Optional Redis ACL username | `""` | @@ -194,8 +194,13 @@ nodeSelector: | `queue.redis.keyPrefix` | Root of every key this release writes (watch cursors, attribution facts, command author records). Give each reverser its own prefix to share one Redis/Valkey between more reversers than `db` can separate. Changing it orphans the previous prefix's keys: cursors cold-replay once, which is safe. Allowed: `[A-Za-z0-9]`, `-`, `_`, `.`, `:` | `gitops-reverser` | | `queue.redis.tls.enabled` | Enable TLS for Redis connection | `false` | | `attribution.enabled` | Run audit ingress and name mirrored-resource commit authors from matching kube-apiserver audit facts | `false` | -| `attribution.ttl` | How long an attribution fact is retained waiting for the matching watch event to join it | `10m` | +| `attribution.transport` | Where attribution facts travel between the audit receiver and the watch side. `redis` appends them to Redis streams and needs `queue.redis.addr`; `memory` keeps them in an in-process ring, which needs no Redis but loses facts on a restart and is **refused at startup with `replicaCount > 1`** | `redis` | +| `attribution.ttl` | How long an attribution fact is retained waiting for the matching watch event to join it. Bounds stream retention and the in-memory index together, and is the horizon a restart replays from | `10m` | | `attribution.grace` | Bounded per-event wait for a matching audit fact before a watch event ships as the committer | `3s` | +| `attribution.maxFactsPerType` | Cap on the facts held in memory for one (audit route, type), evicted oldest-first, so a burst on one noisy type cannot evict every other type's facts | `4096` | +| `attribution.maxFacts` | Cap on the facts held in memory across every type. Must be at least `maxFactsPerType`; overflow evicts from the type holding the most | `65536` | +| `attribution.collectionWindow` | How long after a `deletecollection` a removal in its scope may still be credited to it. It only has to cover audit batching plus clock skew, since the removal is attributed at delete-request time | `30s` | +| `attribution.collectionUIDCap` | How many object UIDs a `deletecollection` fact carries before the set is dropped and the join falls back to scope matching, which is already correct | `10000` | | `attribution.auditRouteAnnotationKey` | Audit-event annotation naming the **audit route** each event belongs to. Empty keeps audit routes named (`/audit-webhook/`). Set it only for a control plane emitting **one shared audit stream** for several logical clusters: it enables the bare `/audit-webhook`, which reads the route per event. A `ClusterProvider` joins a route via `spec.attribution.auditRoute` (default: its own name). An event with no annotation is rejected (counted and logged) and never credited to a fallback | `""` | | `clusterProvider.createDefault` | Render and own a `ClusterProvider` named `default` — the source cluster a `GitTarget` mirrors from when it omits `spec.clusterProviderRef`. The **operator never creates one**, so without this you commit the object yourself. Chart-owned: turning it off makes Helm delete the provider it created, and a `GitTarget` referencing a missing provider is held unready (`ClusterProviderNotFound`). The `quickstart` values never create one | `true` | | `clusterProvider.default.kubeConfig.secretRef.name` | Secret (release namespace) holding a kubeconfig for the rendered `default` provider. Empty means the operator's **own in-cluster** cluster; a name points `default` at a **remote** cluster instead — the name is a convention, not a claim about which cluster it is | `""` | @@ -245,8 +250,8 @@ Audit routes are **named**, including `attribution.auditRouteAnnotationKey` is set, which turns it into the shared-stream endpoint that resolves each event's source cluster from that annotation. The operator extracts a minimal attribution fact from each (auditID, user, verb, resourceVersion, GVR, namespace, name, UID, status, timestamps) -into the Redis attribution index -(populated only when audit attribution is enabled). When a Redis endpoint is configured it also stores +and appends it to the per-type fact log the watch side follows +(written only when audit attribution is enabled). When a Redis endpoint is configured it also stores each GitTarget's watch resume cursors, so reconnects resume a normal watch from the last processed resourceVersion when the apiserver can still serve that history. Object state itself comes from Kubernetes **watch**, not from audit; audit only names the commit author. diff --git a/charts/gitops-reverser/templates/deployment.yaml b/charts/gitops-reverser/templates/deployment.yaml index a771f75e..bf29332e 100644 --- a/charts/gitops-reverser/templates/deployment.yaml +++ b/charts/gitops-reverser/templates/deployment.yaml @@ -80,9 +80,15 @@ spec: {{- if not .Values.queue.redis.tls.enabled }} - --redis-insecure {{- end }} + - --replica-count={{ .Values.replicaCount }} - --author-attribution={{ .Values.attribution.enabled }} + - --author-attribution-transport={{ .Values.attribution.transport }} - --author-attribution-ttl={{ .Values.attribution.ttl }} - --author-attribution-grace={{ .Values.attribution.grace }} + - --author-attribution-max-facts-per-type={{ .Values.attribution.maxFactsPerType }} + - --author-attribution-max-facts={{ .Values.attribution.maxFacts }} + - --author-attribution-collection-window={{ .Values.attribution.collectionWindow }} + - --author-attribution-collection-uid-cap={{ .Values.attribution.collectionUIDCap }} {{- if .Values.attribution.auditRouteAnnotationKey }} - --author-attribution-audit-route-annotation-key={{ .Values.attribution.auditRouteAnnotationKey }} {{- end }} diff --git a/charts/gitops-reverser/templates/validate-redis.yaml b/charts/gitops-reverser/templates/validate-redis.yaml index f5119805..eb24062d 100644 --- a/charts/gitops-reverser/templates/validate-redis.yaml +++ b/charts/gitops-reverser/templates/validate-redis.yaml @@ -1,9 +1,15 @@ {{- /* -Attribution (attributed-author mode) reads and writes audit facts in Redis, so it cannot run -without an endpoint — the controller would fail at startup. Fail the render early with an -actionable message instead. The admission webhook is deliberately NOT gated here: it stays -enabled without Redis and simply no-ops command-author capture. +Attribution needs a fact TRANSPORT, which is not the same as needing Redis. The redis transport +appends facts to Redis streams, so it cannot run without an endpoint — the controller would fail at +startup, so fail the render early with an actionable message instead. The memory transport needs no +endpoint at all, and refusing it here would make a value this chart itself documents unreachable. + +The memory transport's other requirement, a single replica, needs no check here: validate-replica-count.yaml +already refuses replicaCount > 1 for the whole chart, so there is no configuration this could catch. + +The admission webhook is deliberately NOT gated here: it stays enabled without Redis and simply +no-ops command-author capture. */ -}} -{{- if and .Values.attribution.enabled (eq (trim .Values.queue.redis.addr) "") -}} -{{- fail "attribution.enabled=true requires queue.redis.addr: attributed-author mode stores audit facts in Redis. Set queue.redis.addr, or leave attribution.enabled=false." -}} +{{- if and .Values.attribution.enabled (eq .Values.attribution.transport "redis") (eq (trim .Values.queue.redis.addr) "") -}} +{{- fail "attribution.enabled=true with attribution.transport=redis requires queue.redis.addr: the redis transport carries audit facts on Redis streams. Set queue.redis.addr, select attribution.transport=memory to run attribution in-process on a single replica, or leave attribution.enabled=false." -}} {{- end -}} diff --git a/charts/gitops-reverser/values.schema.json b/charts/gitops-reverser/values.schema.json index 5a19b7af..f1fed510 100644 --- a/charts/gitops-reverser/values.schema.json +++ b/charts/gitops-reverser/values.schema.json @@ -275,6 +275,30 @@ "enabled": { "type": "boolean" }, "ttl": { "$ref": "#/$defs/duration" }, "grace": { "$ref": "#/$defs/duration" }, + "transport": { + "type": "string", + "enum": ["redis", "memory"], + "description": "Where attribution facts travel between the audit receiver and the watch side. 'redis' (default) appends them to Redis streams and requires queue.redis.addr; 'memory' keeps them in an in-process ring, which needs no Redis but does not survive a restart and is refused at startup with replicaCount > 1." + }, + "maxFactsPerType": { + "type": "integer", + "minimum": 1, + "description": "Cap on the facts held in memory for one (audit route, group/resource), evicted oldest-first, so a burst on one noisy type cannot evict every other type's facts." + }, + "maxFacts": { + "type": "integer", + "minimum": 1, + "description": "Cap on the facts held in memory across every type. Must be at least maxFactsPerType; overflow evicts from the type holding the most." + }, + "collectionWindow": { + "$ref": "#/$defs/duration", + "description": "How long after a deletecollection a removal in its scope may still be credited to it. It only has to cover audit batching plus clock skew, since the removal is attributed at delete-request time." + }, + "collectionUIDCap": { + "type": "integer", + "minimum": 1, + "description": "How many object uids a deletecollection fact carries before the set is dropped and the join falls back to scope matching, which is already correct." + }, "auditRouteAnnotationKey": { "type": "string", "description": "Audit-event annotation naming the audit route each event belongs to. A ClusterProvider joins a route through spec.attribution.auditRoute, which defaults to its own name. Empty (default) keeps audit routes named (/audit-webhook/); set it only for one shared audit stream carrying several logical clusters, which enables the bare /audit-webhook endpoint." diff --git a/charts/gitops-reverser/values.yaml b/charts/gitops-reverser/values.yaml index 169a21b4..433fc100 100644 --- a/charts/gitops-reverser/values.yaml +++ b/charts/gitops-reverser/values.yaml @@ -204,15 +204,42 @@ queue: # operator runs configured-author (Redis optional — leave queue.redis.addr empty to run without it), and every mirrored-resource commit uses the # configured committer. attribution: - # When true, run the audit webhook ingress and name commit authors from matching audit facts; requires - # a non-empty queue.redis.addr. When false, run configured-author (no audit ingress, commits authored by - # the committer); Redis is optional (leave queue.redis.addr empty to run without it). + # When true, run the audit webhook ingress and name commit authors from matching audit facts. It + # needs a fact transport, which is not the same as needing Redis: `transport: redis` requires a + # non-empty queue.redis.addr, while `transport: memory` needs none and requires replicaCount 1. + # When false, run configured-author (no audit ingress, commits authored by the committer); Redis + # stays optional either way (leave queue.redis.addr empty to run without it). enabled: false - # How long an attribution fact is retained waiting for the matching watch event to join it. + # Where attribution facts travel between the audit receiver and the watch side. + # "redis" (the default) appends them to Redis streams and needs a non-empty queue.redis.addr; it + # is the production choice and the only one that survives a restart or reaches a second replica. + # "memory" keeps them in an in-process ring, so a single-pod install can run attribution with no + # Valkey at all — at the cost that facts do not survive a restart, and it is REFUSED at startup + # with replicaCount > 1, where the audit receiver and the resolver are no longer one process. + transport: "redis" + # How long an attribution fact is retained waiting for the matching watch event to join it. It + # bounds stream retention and the in-memory index together, and doubles as the replay horizon a + # restart warms the index from. ttl: "10m" # Bounded per-event wait for a matching audit fact before a watch event ships as the committer. # Larger values raise attribution hit-rate at the cost of commit latency. grace: "3s" + # Caps on the facts held in memory. Per-type is the primary because it is the fair one: a burst on + # one noisy type — a deletecollection over ten thousand objects, a large rollout — must not evict + # every other type's facts. The total bounds the pod with a number that does not scale with how + # many types happen to be watched; overflow evicts from the type holding the most. Evictions are + # counted on attribution_fact_index_evictions_total{reason}. + maxFactsPerType: 4096 + maxFacts: 65536 + # How long after a deletecollection a removal in its scope may still be credited to it. It only has + # to cover audit batching plus clock skew — the removal is attributed at delete-REQUEST time, so + # finalizers do not stretch it. Longer widens the risk of crediting an unrelated delete to the + # collection's actor. + collectionWindow: "30s" + # How many object uids a deletecollection fact carries before the set is dropped and the join falls + # back to scope matching. The fallback is already correct, so this only decides how often the + # precise path is taken; drops are counted on attribution_collection_without_uidset_total{reason}. + collectionUIDCap: 10000 # Audit-event annotation naming the AUDIT ROUTE each event belongs to. Audit routes are normally # NAMED (/audit-webhook/, including /audit-webhook/default), and this is empty. # Set it only for a control plane that emits ONE shared audit stream for several logical diff --git a/cmd/main.go b/cmd/main.go index 33095c5d..a0f2a361 100644 --- a/cmd/main.go +++ b/cmd/main.go @@ -232,15 +232,25 @@ func main() { var ( auditRunnable *auditServerRunnable auditCertWatcher *certwatcher.CertWatcher - attributionIndex *queue.AttributionIndex ) switch { case cfg.authorAttribution: - attributionIndex = redisStore.AttributionIndex(cfg.attributionFactTTL) + // The transport is the only thing that differs between the two modes. Everything above it — + // the index, the waiter registry, the subscription set, the resolver — has one implementation + // and never learns which transport it was handed. + transport := buildFactTransport(cfg, redisStore) + factIndex := queue.NewFactIndex(queue.FactIndexConfig{ + TTL: cfg.attributionFactTTL, + MaxFactsPerType: cfg.attributionMaxFactsPerType, + MaxFactsTotal: cfg.attributionMaxFacts, + CollectionWindow: cfg.attributionCollectionWindow, + Log: ctrl.Log.WithName("attribution-index"), + }) auditHandler, err := webhookhandler.NewAuditHandler(webhookhandler.AuditHandlerConfig{ MaxRequestBodyBytes: cfg.auditMaxRequestBodyBytes, - FactRecorder: attributionIndex, + FactPublisher: transport, + CollectionUIDCap: cfg.attributionCollectionUIDCap, // Empty leaves the bare /audit-webhook endpoint disabled (400); set, it demultiplexes a // shared stream per event by this annotation. AuditRouteAnnotationKey: cfg.auditRouteAnnotationKey, @@ -252,14 +262,26 @@ func main() { fatalIfErr(initErr, "unable to initialize audit ingress server") fatalIfErr(mgr.Add(auditRunnable), "unable to add audit ingress server runnable") + // The follower runs for the life of the process, reading whatever the watches are currently + // subscribed to. It follows nothing until a watch acquires a stream, and a newly followed + // stream is replayed from the TTL horizon, so a watch that starts late still finds a warm index. + fatalIfErr(mgr.Add(factFollowerRunnable{index: factIndex, follower: transport}), + "unable to add attribution fact follower runnable") + + watchMgr.FactStreams = factIndex.Streams() watchMgr.AuthorResolver = watch.NewAuthorResolver( - attributionIndex, + factIndex, cfg.attributionGrace, ctrl.Log.WithName("attribution"), ) setupLog.Info("author attribution enabled: matched audit facts name the commit author", - "redisAddr", cfg.redisAddr, "grace", cfg.attributionGrace.String(), + "transport", cfg.attributionTransport, "redisAddr", cfg.redisAddr, + "grace", cfg.attributionGrace.String(), "factTTL", cfg.attributionFactTTL.String(), "auditRouteAnnotationKey", cfg.auditRouteAnnotationKey) + if cfg.attributionTransport == transportMemory { + setupLog.Info("attribution facts travel in process memory: they do not survive a restart, so " + + "events in flight across one lose their author, and this mode requires a single replica") + } if cfg.auditRouteAnnotationKey == "" { setupLog.Info("audit routes are named: post to /audit-webhook/, where the route is " + "ClusterProvider.spec.attribution.auditRoute and defaults to the provider's own name; " + @@ -396,12 +418,23 @@ type appConfig struct { redisKeyPrefix string redisInsecure bool authorAttribution bool + attributionTransport string attributionFactTTL time.Duration attributionGrace time.Duration + attributionMaxFactsPerType int + attributionMaxFacts int + attributionCollectionWindow time.Duration + attributionCollectionUIDCap int auditRouteAnnotationKey string - branchBufferMaxBytes int64 - sensitiveResources types.SensitiveResourcePolicy - sshHostKeys git.SSHHostKeyConfig + // replicaCount is how many replicas of this Deployment run. The process cannot observe that + // itself, so the chart templates it in from .Values.replicaCount. It exists for exactly one + // check: the in-memory fact transport only works when the audit receiver and the resolver are + // the same process, and that has to fail loudly rather than degrade into silent attribution + // loss. + replicaCount int + branchBufferMaxBytes int64 + sensitiveResources types.SensitiveResourcePolicy + sshHostKeys git.SSHHostKeyConfig // sourceClusterQPS / sourceClusterBurst bound the rate at which the operator talks to a // source cluster reached through a GitTarget.spec.kubeConfig. A remote is reached over a // network the in-cluster config is not, so it carries client-side throttling by default. @@ -476,10 +509,11 @@ func parseFlagsWithArgs(fs *flag.FlagSet, args []string) (appConfig, error) { "Idle timeout for the audit ingress HTTPS server (duration string; default 60s).") fs.StringVar(&cfg.redisAddr, "redis-addr", "valkey:6379", "Redis/Valkey address (host:port). Holds each GitTarget's watch resume cursors (state continuity) "+ - "and, when author attribution is enabled, the attribution facts. Leave empty to run without "+ - "Redis: watches cold-replay on restart instead of resuming. Required by "+ - "--author-attribution=true. --admission-webhook still runs without it, but command-author "+ - "capture is a no-op: CommitRequests claim no actor.") + "and, with --author-attribution-transport=redis, the attribution fact streams. Leave empty to "+ + "run without Redis: watches cold-replay on restart instead of resuming, and attribution needs "+ + "--author-attribution-transport=memory. Required by --author-attribution-transport=redis when "+ + "attribution is on. --admission-webhook still runs without it, but command-author capture is a "+ + "no-op: CommitRequests claim no actor.") fs.BoolVar(&cfg.authorAttribution, "author-attribution", true, "Name the real actor (human or service account) who caused each change as the Git commit author, "+ "resolved from matching audit facts; this runs the audit webhook ingress (default true). When "+ @@ -502,9 +536,37 @@ func parseFlagsWithArgs(fs *flag.FlagSet, args []string) (appConfig, error) { "Connect to Redis over plain TCP instead of TLS (default false; TLS). Redis carries each "+ "GitTarget's watch cursors and, when attribution is on, the audit facts — prefer TLS. Set "+ "this only for a trusted in-cluster Redis/Valkey that does not serve TLS.") + fs.StringVar(&cfg.attributionTransport, "author-attribution-transport", transportRedis, + "Where attribution facts travel between the audit receiver and the watch side: \"redis\" (default) "+ + "appends to Redis streams, \"memory\" to an in-process ring. Redis is the production choice and "+ + "the only one that survives a restart or reaches a second replica; memory is for the single-pod "+ + "install where running a Valkey StatefulSet to name commit authors is out of proportion, and it "+ + "is refused with more than one replica. With \"memory\", --redis-addr may be empty.") fs.DurationVar(&cfg.attributionFactTTL, "author-attribution-ttl", queue.DefaultAttributionFactTTL, - "How long an attribution fact is retained waiting for the matching watch event to join it "+ - "(duration string; default 10m).") + "How long an attribution fact is retained waiting for the matching watch event to join it. It "+ + "bounds stream retention and the in-memory index together, and doubles as the replay horizon a "+ + "restart warms the index from (duration string; default 10m).") + fs.IntVar(&cfg.attributionMaxFactsPerType, "author-attribution-max-facts-per-type", + queue.DefaultFactIndexMaxFactsPerType, + "Cap on the attribution facts held in memory for one (audit route, group/resource), evicted "+ + "oldest-first (default 4096). It is the fair cap: a burst on one noisy type must not evict every "+ + "other type's facts. Evictions are counted on attribution_fact_index_evictions_total{reason}.") + fs.IntVar(&cfg.attributionMaxFacts, "author-attribution-max-facts", queue.DefaultFactIndexMaxFactsTotal, + "Cap on the attribution facts held in memory across every type (default 65536), so the pod's "+ + "memory is bounded by a number that does not scale with how many types happen to be watched. "+ + "Overflow evicts from the type holding the most.") + fs.DurationVar(&cfg.attributionCollectionWindow, "author-attribution-collection-window", + queue.DefaultFactCollectionWindow, + "How long after a deletecollection a removal in its scope may still be credited to it (duration "+ + "string; default 30s). It only has to cover audit batching plus clock skew: the removal is "+ + "attributed at delete-REQUEST time, so finalizers do not stretch it. Longer widens the risk of "+ + "crediting an unrelated delete to the collection's actor.") + fs.IntVar(&cfg.attributionCollectionUIDCap, "author-attribution-collection-uid-cap", + queue.DefaultCollectionUIDCap, + "How many object uids a deletecollection fact may carry before the set is dropped and the join "+ + "falls back to scope matching (default 10000). The fallback is already correct, so this only "+ + "decides how often the precise path is taken; drops are counted on "+ + "attribution_collection_without_uidset_total{reason}.") fs.DurationVar(&cfg.attributionGrace, "author-attribution-grace", watch.DefaultAttributionGraceWindow, "Bounded per-event wait for a matching audit fact to arrive before a watch event ships as the "+ "configured committer (duration string; default 3s). Larger values raise attribution hit-rate "+ @@ -517,6 +579,10 @@ func parseFlagsWithArgs(fs *flag.FlagSet, args []string) (appConfig, error) { "provider's own name). An event carrying no annotation is rejected (counted and logged) rather "+ "than credited to a fallback. Empty (the default) leaves the bare endpoint disabled, and every "+ "producer must post to /audit-webhook/.") + fs.IntVar(&cfg.replicaCount, "replica-count", 1, + "How many replicas of this Deployment run (default 1). The process cannot see this itself, so "+ + "the chart passes it in; it exists to refuse --author-attribution-transport=memory with more "+ + "than one replica, where the audit receiver and the resolver are no longer the same process.") branchBufferMaxSizeStr := os.Getenv("BRANCH_BUFFER_MAX_SIZE") if branchBufferMaxSizeStr == "" { branchBufferMaxSizeStr = defaultBranchBufferMaxSizeStr @@ -613,6 +679,15 @@ func bindServerCertFlags( fmt.Sprintf("The name of the %s key file.", component)) } +// transportRedis and transportMemory are the two values --author-attribution-transport accepts. +// The choice is explicit rather than inferred from an empty --redis-addr: an empty address means +// attribution is off today, and quietly making it mean "attribution over an in-memory transport" +// would change existing installs by inference. +const ( + transportRedis = "redis" + transportMemory = "memory" +) + func validateAuditConfig(cfg appConfig) error { if cfg.attributionGrace < 0 { return fmt.Errorf("author-attribution-grace must be >= 0, got %s", cfg.attributionGrace) @@ -623,6 +698,9 @@ func validateAuditConfig(cfg appConfig) error { if cfg.redisDB < 0 { return fmt.Errorf("redis-db must be >= 0, got %d", cfg.redisDB) } + if err := validateAttributionTransport(cfg); err != nil { + return err + } // The annotation key only has a receiver to configure when the audit ingress is running at all; // silently ignoring it would look like annotation routing was enabled when nothing serves it. if cfg.auditRouteAnnotationKey != "" && !cfg.authorAttribution { @@ -631,11 +709,20 @@ func validateAuditConfig(cfg appConfig) error { "without it there is no audit ingress to route") } if strings.TrimSpace(cfg.redisAddr) == "" { - if cfg.authorAttribution { - return errors.New("redis-addr is required when author-attribution is enabled") + // Attribution no longer implies Redis. It needs a fact transport, and the in-memory one is a + // supported configuration rather than a rejected one — which is the whole reason the transport + // is selectable. Redis stays required for the transport that uses it, and for the admission + // webhook's command-author capture, which has no in-memory counterpart. + if cfg.authorAttribution && cfg.attributionTransport == transportRedis { + return errors.New( + "redis-addr is required when author-attribution is enabled with " + + "author-attribution-transport=redis; set an address, or select " + + "author-attribution-transport=memory to run attribution in-process on a single replica") + } + if !cfg.authorAttribution { + // Configured-author mode with no Redis: watches cold-replay on restart, attribution is off. + return nil } - // Configured-author mode with no Redis: watches cold-replay on restart, attribution is off. - return nil } if !cfg.authorAttribution { // Configured-author mode with Redis configured: the audit ingress server is not started, so its @@ -663,6 +750,49 @@ func validateAuditConfig(cfg appConfig) error { return nil } +// validateAttributionTransport checks the transport selection and the caps that configure the index +// behind it. The replica gate is the one that matters: the in-memory transport only works when the +// audit receiver and the resolver are the same process, so under more than one replica an audit POST +// answered by pod A leaves pod B's watch with no fact at all. That must fail loudly at startup +// rather than degrade into commits silently authored "attribution unresolved". +func validateAttributionTransport(cfg appConfig) error { + switch cfg.attributionTransport { + case transportRedis, transportMemory: + default: + return fmt.Errorf("author-attribution-transport must be %q or %q, got %q", + transportRedis, transportMemory, cfg.attributionTransport) + } + if !cfg.authorAttribution { + return nil + } + if cfg.attributionTransport == transportMemory && cfg.replicaCount > 1 { + return fmt.Errorf( + "author-attribution-transport=memory requires a single replica, got replica-count=%d: the "+ + "in-memory transport only carries facts within one process, so an audit event answered by "+ + "one replica can never name the author for a watch running on another. Use "+ + "author-attribution-transport=redis for more than one replica", cfg.replicaCount) + } + if cfg.attributionMaxFactsPerType <= 0 { + return fmt.Errorf("author-attribution-max-facts-per-type must be > 0, got %d", cfg.attributionMaxFactsPerType) + } + if cfg.attributionMaxFacts <= 0 { + return fmt.Errorf("author-attribution-max-facts must be > 0, got %d", cfg.attributionMaxFacts) + } + if cfg.attributionMaxFacts < cfg.attributionMaxFactsPerType { + return fmt.Errorf( + "author-attribution-max-facts (%d) must be >= author-attribution-max-facts-per-type (%d); "+ + "a total cap below the per-type cap makes the per-type cap unreachable", + cfg.attributionMaxFacts, cfg.attributionMaxFactsPerType) + } + if cfg.attributionCollectionWindow <= 0 { + return fmt.Errorf("author-attribution-collection-window must be > 0, got %s", cfg.attributionCollectionWindow) + } + if cfg.attributionCollectionUIDCap <= 0 { + return fmt.Errorf("author-attribution-collection-uid-cap must be > 0, got %d", cfg.attributionCollectionUIDCap) + } + return nil +} + func validateAdmissionWebhookConfig(cfg appConfig) error { if !cfg.admissionWebhookEnabled { return nil @@ -735,6 +865,32 @@ func buildMetricsServerOptions( return opts, metricsCertWatcher } +// buildFactTransport builds the selected attribution fact transport. Both sides of the seam come +// from one object: the audit receiver publishes into it and the index follows it, which is exactly +// the property the in-memory mode depends on and the reason it is refused with a second replica. +// +// The Redis branch cannot be reached with a nil store: validateAuditConfig rejects an empty +// --redis-addr with attribution on and the Redis transport selected. +func buildFactTransport(cfg appConfig, redisStore *queue.RedisStore) queue.FactTransport { + if cfg.attributionTransport == transportMemory { + return queue.NewMemoryFactStream(queue.MemoryFactStreamConfig{TTL: cfg.attributionFactTTL}) + } + return redisStore.FactStream(queue.RedisFactStreamConfig{TTL: cfg.attributionFactTTL}) +} + +// factFollowerRunnable runs the fact index's follow loop under the manager, so it starts with the +// process and stops with it. The loop returns only when the context ends: a transport failure is +// retried, because a follower that gave up would leave attribution silently dead for the life of +// the process. +type factFollowerRunnable struct { + index *queue.FactIndex + follower queue.FactFollower +} + +func (r factFollowerRunnable) Start(ctx context.Context) error { + return r.index.Run(ctx, r.follower) +} + type auditServerRunnable struct { server *http.Server tlsEnabled bool diff --git a/cmd/main_redis_flags_test.go b/cmd/main_redis_flags_test.go index c2487750..e04b8c76 100644 --- a/cmd/main_redis_flags_test.go +++ b/cmd/main_redis_flags_test.go @@ -20,10 +20,20 @@ func TestParseFlags_RedisAddrIsOnlyRequiredByAttribution(t *testing.T) { args []string wantErr string }{ - "attribution needs redis": { + "attribution over the redis transport needs redis": { args: []string{"--redis-addr=", "--author-attribution=true"}, wantErr: "redis-addr is required when author-attribution is enabled", }, + // The narrowed rule, and the whole reason the transport is selectable: attribution needs a + // fact TRANSPORT, not Redis specifically. In-memory plus an empty address is a supported + // configuration rather than a rejected one, or the mode would be unreachable. + "attribution over the memory transport runs without redis": { + args: []string{ + "--redis-addr=", "--author-attribution=true", + "--author-attribution-transport=memory", + "--audit-insecure", + }, + }, // The webhook is failurePolicy: Ignore by design and the controller is the real gate, // so running it without Redis is a supported, degraded mode — not a usage error. // (--admission-webhook-cert-path has no default and is required whenever the webhook @@ -54,3 +64,63 @@ func TestParseFlags_RedisAddrIsOnlyRequiredByAttribution(t *testing.T) { }) } } + +// TestParseFlags_AttributionTransportSelection pins the guard rails around the transport choice. +// The replica gate is the one that matters: the in-memory transport carries facts only within one +// process, so under more than one replica an audit POST answered by pod A leaves pod B's watch with +// nothing to join. That has to be a startup error, because the alternative is commits that are +// silently authored "attribution unresolved" and an operator with no way to tell why. +func TestParseFlags_AttributionTransportSelection(t *testing.T) { + tests := map[string]struct { + args []string + wantErr string + }{ + "memory with one replica is fine": { + args: []string{"--author-attribution-transport=memory", "--replica-count=1", "--audit-insecure"}, + }, + "memory with more than one replica is refused": { + args: []string{"--author-attribution-transport=memory", "--replica-count=2"}, + wantErr: "author-attribution-transport=memory requires a single replica", + }, + // Redis carries facts between processes, so the replica count is not its business. + "redis with more than one replica is not this check's business": { + args: []string{"--author-attribution-transport=redis", "--replica-count=2", "--audit-insecure"}, + }, + "an unknown transport is refused": { + args: []string{"--author-attribution-transport=kafka"}, + wantErr: `author-attribution-transport must be "redis" or "memory", got "kafka"`, + }, + // With attribution off there is no transport in play, so an unknown value is still a typo + // worth catching, but the replica pairing is not checked. + "memory with many replicas is moot when attribution is off": { + args: []string{ + "--author-attribution=false", "--author-attribution-transport=memory", "--replica-count=3", + }, + }, + "a total cap below the per-type cap makes the per-type cap unreachable": { + args: []string{ + "--author-attribution-max-facts-per-type=4096", "--author-attribution-max-facts=1024", + }, + wantErr: "must be >= author-attribution-max-facts-per-type", + }, + "a zero collection window is refused": { + args: []string{"--author-attribution-collection-window=0"}, + wantErr: "author-attribution-collection-window must be > 0", + }, + "a zero uid cap is refused": { + args: []string{"--author-attribution-collection-uid-cap=0"}, + wantErr: "author-attribution-collection-uid-cap must be > 0", + }, + } + + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + _, err := parseArgs(t, tc.args...) + if tc.wantErr != "" { + require.ErrorContains(t, err, tc.wantErr) + return + } + require.NoError(t, err) + }) + } +} diff --git a/cmd/mutation-capture-lab/main.go b/cmd/mutation-capture-lab/main.go index cf54ffdc..5d142b67 100644 --- a/cmd/mutation-capture-lab/main.go +++ b/cmd/mutation-capture-lab/main.go @@ -121,6 +121,12 @@ func buildServers( auditMux := http.NewServeMux() auditMux.Handle("/audit-webhook", audit) + // Audit routes are NAMED in the e2e cluster: its bootstrap kubeconfig posts to + // /audit-webhook/ (the "default" ClusterProvider), because the bare + // path is the product's shared annotation-routed endpoint. The lab records what + // arrives regardless of which source it was addressed to, so it takes the whole + // subtree — an exact-path-only mux 404s every event the cluster sends. + auditMux.Handle("/audit-webhook/", audit) return []server{ {name: "admission", certDir: cfg.admissionCert, srv: &http.Server{ diff --git a/docs/INDEX.md b/docs/INDEX.md index 0faa1084..1f403bbc 100644 --- a/docs/INDEX.md +++ b/docs/INDEX.md @@ -71,11 +71,14 @@ Fifteen other open items: |---|---| | [`open-asks-priority.md`](design/open-asks-priority.md) | three backlogs are open at once — the gitops-api consumer asks, the maintainer review's unbuilt block (F6, F9, F10), and the config-surface proposal (B1–B6) — and they overlap. Merges them into one ordered queue under four stated tests, and makes one design call against what was asked: **delete Option C sibling inference** rather than ship an off-switch for it, because it lets a human's edit to the repository change the operator's behaviour with nothing in status recording the move, its central guard has already failed once by cascading, and the explainability its own spec made mandatory was never built. That answers the namespace-leak ask by removal, and means `spec.placement.mode` is never built. Open: whether the removal ships with an Event on the first fall-back to canonical | | [`docs-linting.md`](design/docs-linting.md) | how to mechanize [`style-guide.md`](style-guide.md) with markdownlint-cli2 and Vale. Both are wired into `task lint`, gated on the files [`.docs-lint-scope`](../.docs-lint-scope) lists rather than the whole tree: 102 of 174 files fail markdownlint and 148 of 174 fail Vale, so the two backlogs need different gates. Open: how the scope list grows to cover the tree, the `MD013` limit, and whether `AGENTS.md` and the chart READMEs are in scope | +| [`attribution-removal-wait-options.md`](design/attribution-removal-wait-options.md) | a removal now waits for evidence about the DELETION rather than accepting the object's last write, which stopped it naming whoever last edited the object as the author of a deletion they did not perform. Enumerates the eight situations a resolution can be in and shows the cost is concentrated in exactly one: a removal for which no delete fact will ever arrive (a graceful pod delete, a status-only removal, a type the audit policy skips) spends the whole grace to return the answer it had at t=0, measured at ~3.1s against ~70ms when evidence is present. Prices five options against that, and recommends a per-route watermark — stop waiting once the fact stream has demonstrably moved past this event — over a second timeout flag whose right value lives in the API server's config rather than ours. Open: the decision, and how common the case is outside the e2e suite | +| [`attribution-metrics-proposal.md`](design/attribution-metrics-proposal.md) | a phased attribution metric surface, revised after review cut an earlier draft of thirteen new families down to a first release that covers health and the unseen loss paths. Splits `result` into `tier` and `actor_kind` (how `commits_total` already models it) and `weak` into `latest` and `resource_version`, taking the break in the release that has broken `result` anyway. Adds watch-queue delay, follower error and last-success health, a `no_attribution_fact` outcome on the existing bounded audit vocabulary, and a decode-error counter for the one loss path with no symptom at all: both transports discard an undecodable stream entry and advance past it with no log and no metric. Records what the first draft got wrong and why, including a proposed series that would have been permanently zero and a gauge that would have counted registrations rather than blocked resolvers. Its Phase 1 is now Phase 1 of [`metrics-observability-plan.md`](design/metrics-observability-plan.md), which absorbed the surface and records the drift it had accumulated; this stays as the reasoning trail. **Phase 1 has shipped** — the migration is in [`UPGRADING.md`](UPGRADING.md). Open: nothing structural | +| [`attribution-publish-and-join.md`](design/attribution-publish-and-join.md) | the reference for what attribution's two halves each do, exactly: the publish side that turns one audit event into zero or one fact and files it under the keys it happens to have, and the join side that walks the tiers strongest-first to name an author for a watch event. A flowchart per half, the tier table, and the two rules that are easy to miss (a removal never answers with a write fact without looking further; an exact-capable event may never fall through to the removal tiers). Also answers whether anything special-cases a type: nothing does, every branch is on the verb or on which fields are present, and the two ConfigMap deletes in the corpus — one answered with the object, one with a `Status` — are the standing argument that a type-based rule would be unsound | +| [`attribution-branch-findings.md`](design/attribution-branch-findings.md) | what the attribution switchover's loose ends turned out to be, measured rather than reasoned. The mutation lab was serving `/audit-webhook` as an exact path while the cluster posts to the named `/audit-webhook/default`, so every audit event 404'd and every audit-carrying scenario timed out — a routing mismatch that reads exactly like a broken cluster. With it fixed, the corpus answers the aggregated-API removal question: a proxied delete is audited with a name but **no uid and no resourceVersion**, so the exact and latest tiers can never match it, and a proxied `deletecollection` returns **no response body**, so its fact carries no uid set and the join must fall back to scope. Separates the missing name from the missing body — a `generateName` create recovers both from the response object, an aggregated write has nothing to recover from — and prices a name tier against accepting that aggregated types are collection-only. Open: the tier decision, and whether a CommitRequest missing the window of the write it follows by two seconds is new on this branch | | [`attribution-fact-identity.md`](design/attribution-fact-identity.md) | several `ClusterProvider`s may name one physical cluster, but a kube-apiserver posts audit to one route, so only one of those names is ever fed and every other one authors `unknown (attribution unresolved)`. Proposes a declared `spec.attribution.auditRoute` that partitions the facts instead of `metadata.name`, so several providers can share one cluster's facts while cloned clusters stay separate, ingestion loses its last Kubernetes read, and a misrouted provider becomes loud. Renames the key infix and the annotation-key flag to the same word | -| [`attribution-fact-stream.md`](design/attribution-fact-stream.md) | the chosen replacement for the attribution keyspace: the audit receiver appends one batched entry per type to a per-`(route, group/resource)` Redis **stream**, every process follows only the types it watches, and the facts live in one bounded, TTL'd in-memory index per process. Deletes the per-key `SET`/`GET` lookup and the poll loop; keeps the blocking grace window, commit order, and the one-fact-serves-every-GitTarget property. Argues against plain publish and subscribe (silent, undetectable loss on every restart, reconnect, and new watch, once the keys are gone) and against a per-GitTarget index. Also deletes the `deletecollection` expander: one collection fact that every removal in its (type, namespace, selector, window) scope joins, ranked below every per-object fact, which removes the response-body parsing and starts resolving the aggregated-API and metadata-only cases that degrade to committer today. Chosen partly to unblock HA: audit POST and watch shard land on different replicas by construction, and a resumable stream survives the rollouts an HA install spends its life in. Open: cap granularity, entry size, and whether a replica warms its index before taking over a watch | | [`attribution-wait-poll-vs-push.md`](design/attribution-wait-poll-vs-push.md) | **superseded by the above, kept as the reasoning trail.** a watch event needs its author before it can be routed, and the audit fact naming that author may not have arrived yet, so `ResolveAuthor` polls Redis every 150ms for up to three seconds on the watch shard's own goroutine. Separates the wait (forced: two unordered deliveries out of one kube-apiserver, and the commit window groups by author) from the poll (a choice). Answers which of the two fires first: the watch, nearly always, because audit delivery is batched by the apiserver while the watch is streamed, so the first lookup is a near-guaranteed miss and the loop runs to completion on every attributable event. Six options priced against that, from shifting the first check to the delivery floor and a circuit breaker for an audit route that has never resolved anything, through a Redis publish and subscribe, to the reassembly-buffer design that stops blocking the watch shard. Open: whether the wait population is dominated by resolved-late (favors publish and subscribe) or never-resolved (favors the buffer), plus a proposed per-scenario timing report from the mutation-capture lab | | [`watch-and-catalog-architecture.md`](design/watch-and-catalog-architecture.md) | the target three-layer watch model — **needs a human call before building** | -| [`metrics-observability-plan.md`](design/metrics-observability-plan.md) | the watch-stage metrics do not exist yet | +| [`metrics-observability-plan.md`](design/metrics-observability-plan.md) | the canonical metrics plan, reconciled to the code after the fact-stream switchover and now carrying the attribution surface from [`attribution-metrics-proposal.md`](design/attribution-metrics-proposal.md). Reads the product as one pipeline — watch events arrive, and are processed into commits — and maps a metric to each stage. The attribution join is built and correctly labelled; **watch ingestion, shard queue delay, and the relevance filter are still dark**. **Phase 1 — the attribution relabel plus the loss-path counters — has shipped**; Phase 2 is the watch stage, Phase 3 the filter and push health, Phase 4 the dashboard and alerts. Open: Phases 2-4, and the dashboard JSON is deliberately not written until the watch families exist | | [`reconcile-triggering.md`](design/reconcile-triggering.md) | which controllers still fail to wake up | | [`multi-source-audit-ingress-hardening.md`](design/multi-source-audit-ingress-hardening.md) | how independent sources authenticate to a named audit route, when annotation routing is trustworthy, and how multi-provider ingestion remains fair | | [`release-image-reuse-plan.md`](design/release-image-reuse-plan.md) | PRs 2–5 unstarted | @@ -102,11 +105,22 @@ Five more ideas sit beside them. ## History — [`finished/`](finished/) -Twenty-one shipped plans and closed investigations. **Nothing here binds.** Read one +Twenty-two shipped plans and closed investigations. **Nothing here binds.** Read one only when you want to know *why* something is the way it is; the answer to *what it is* always lives in `spec/`. The newest is +[`attribution-fact-stream.md`](finished/attribution-fact-stream.md): why attribution facts stopped +being a keyspace the watch side polls and became a per-type log it follows into a bounded in-memory +index. It deleted the `SET`/`GET` fact keys, the 150ms poll loop, and the `deletecollection` +expander, replacing the expander with one collection fact that every removal in its scope joins by +uid membership or by scope. That last part is a capability gain rather than a like-for-like swap: a +collection delete the API server sent no response body for used to lose its author entirely. +`exact_deletecollection_item` is replaced by `collection_uid` and `collection_scope`, and +`--author-attribution-transport=memory` runs attribution with no Redis on a single replica. +Shipped as #283, #284, #286 and #287. + +Before it, [`analyzer-consumer-contract-asks.md`](finished/analyzer-consumer-contract-asks.md): why a refusal carries whether it can be solved and by whom, why the analyzer's report is a KRM document that names the build that produced it, and why `ResourceIdentifier.Key()` is a diff --git a/docs/UPGRADING.md b/docs/UPGRADING.md index 073784ab..3dfe078d 100644 --- a/docs/UPGRADING.md +++ b/docs/UPGRADING.md @@ -7,6 +7,102 @@ guidance that the changelog's breaking-change entries link to. We are pre-1.0, so breaking changes bump the **minor** version (release-please is configured with `bump-minor-pre-major`) rather than the major. Read the relevant entry before upgrading across it. +## 0.41.0 — attribution facts travel on a selectable transport, and Redis is no longer implied + +Attribution stopped meaning Redis. The audit receiver appends its facts to a per-type **stream**, +the watch side follows the streams for the types it watches into a bounded in-process index, and +which stream implementation carries them is a choice: + +| `attribution.transport` (`--author-attribution-transport`) | Store | When | +|---|---|---| +| `redis` — the default, and the previous behaviour | Redis streams, per type | any install; the only one that survives a restart | +| `memory` | an in-process ring | a single-replica install with no Valkey; every fact is lost on restart, by design | + +**A default upgrade needs no action.** `redis` is the default, `queue.redis.addr` keeps its meaning, +and the same events resolve to the same authors. + +Two things to know if you look closely: + +- **Facts in flight across the upgrade lose their author.** The v1 fact keyspace is gone, and the + streams are a new keyspace rather than a migration of it, so a fact written by the old version is + not read by the new one. A watch event whose fact was written moments before the restart resolves + `absent` and its commit is authored `unknown (attribution unresolved)`. The window is one fact TTL + at most, and mirroring is unaffected — attribution changes the author, never the state. +- **`attribution.transport=memory` requires `queue.redis.addr` to be empty-able, and a single + replica.** The chart refuses `redis` without an address at render time, and the binary refuses + `memory` with `--replica-count` above 1 at startup, because the audit receiver and the resolver + must be one process for in-process facts to be visible at all. Redis is still required for the + admission webhook's command-author capture, which has no in-process counterpart. + +New tuning knobs, all with behaviour-preserving defaults: `attribution.maxFactsPerType` (4096), +`attribution.maxFacts` (65536), `attribution.collectionWindow` (30s), `attribution.collectionUIDCap` +(10000). They bound the in-process index and the collection join; see +[configuration.md](configuration.md). + +## 0.41.0 — the attribution metrics are relabelled and partly renamed (breaking for queries) + +The `result` label is **gone** from `gitopsreverser_attribution_resolutions_total` and +`gitopsreverser_attribution_resolution_wait_seconds`. It crammed two orthogonal questions into one +value — which evidence answered, and who it named — and hid a third, so it is replaced by two labels +on the counter and a different second label on the histogram. Four metric families are renamed in the +same change, while the surface is moving. + +Nothing else in the pipeline changes: the same events resolve to the same authors, and the same +commits are written. Only the names a query selects on change. + +| Old | New | +| --- | --- | +| `attribution_resolutions_total{result}` | `attribution_resolutions_total{tier, actor_kind}` | +| `attribution_resolution_wait_seconds{result}` | `attribution_resolution_wait_seconds{tier, event_kind}` | +| `attribution_fact_events_total{op}` | `attribution_facts_total{op}` | +| `attribution_fact_index_size` | `attribution_fact_index_entries` | +| `attribution_collection_degraded_total{reason}` | `attribution_collection_without_uidset_total{reason}` | + +`attribution_fact_index_evictions_total{reason}` and `attribution_fact_stream_gaps_total{stream}` +are unchanged. + +The values move too. `tier` carries what `result` said about evidence, and `weak` splits in two, +because it covered two different kinds of it: + +| Old `result` | New | +| --- | --- | +| `exact_user` | `tier="exact"`, `actor_kind="user"` | +| `exact_serviceaccount` | `tier="exact"`, `actor_kind="serviceaccount"` | +| `weak` (a UID-latest match) | `tier="latest"` | +| `weak` (the RV-only escape hatch) | `tier="resource_version"` | +| `collection_uid`, `collection_scope`, `name`, `absent` | `tier=` the same value | + +`actor_kind` is `user`, `serviceaccount`, or `none`, the vocabulary +`gitopsreverser_commits_total{author_kind}` already uses, and it is available on **every** tier +rather than on the exact one alone. `event_kind` on the wait histogram is `write` or `removal`: a +removal holds a fallback and keeps waiting for evidence about the deletion where a write does not, +so `{event_kind="removal"}` is the distribution `--author-attribution-grace` is tuned from. + +**Rewrite match coverage as `tier!="absent"`.** A query kept as `result=~"exact_.*"` — or ported to +`tier=~"exact.*"` — reads the collection and name tiers as misses, and those tiers named an actor. + +Four signals are new in the same release, so a query is worth writing against them at the same time: + +- `attribution_fact_stream_decode_errors_total{transport}` — an entry the follower refuses is + skipped and its facts lost, and this is the one loss path with no other symptom. It covers both a + payload that is not JSON and one that breaks the fact contract by naming nobody (`author` is + required and non-empty). +- `attribution_fact_follower_errors_total{transport}` and + `attribution_fact_follower_last_success_timestamp_seconds`. Alert on **both** arms of the + timestamp: it is not published until the follower's first successful read, so + `time() - ` returns no series for a follower wedged since startup rather than a large + number. The expression that covers that case is in the field guide. +- `attribution_transport_info{transport}` — an info gauge, always 1, naming the transport in force. + It is a legend rather than a threshold: a burst of unresolved commits after a restart is expected + under `memory` and a bug under `redis`. + +`gitopsreverser_audit_events_total` also gains a `no_attribution_fact` outcome in the `dropped` +category, for an accepted event that produces no fact — a population previously counted `queued`. +The `category="error"` invariant is unaffected. + +Every one of these is documented in +[interpreting-metrics.md](interpreting-metrics.md#audit-attribution-optional). + ## `summary.fleetRoot` is gone from the analyzer report (breaking JSON change) `manifest-analyzer --mode scan-repo` no longer emits `status.summary.fleetRoot`, the text diff --git a/docs/architecture.md b/docs/architecture.md index 30c726ee..1e6e355a 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -50,15 +50,18 @@ Every write to that branch goes through the worker's single event loop and commi **Redis/Valkey is optional but advised.** The default configured-author mode runs without it: a plain `helm install` comes up healthy and watches cold-replay on restart. When an endpoint is configured, Redis stores watch resume cursors (warm restarts) and the small coordination records used by -attribution, CommitRequest author capture, and HA. Attributed-author mode requires Redis, and HA will -require it as the shared store across replicas. +CommitRequest author capture and HA. Attribution no longer requires it on its own: its facts travel +on a selectable transport, Redis Streams by default and an in-process ring with +`--author-attribution-transport=memory`, which is refused with more than one replica. HA will require +Redis as the shared store across replicas. **Audit is an optional attribution lookup.** When attribution is enabled, kube-apiserver posts audit events to `/audit-webhook/` (or a configured annotation-routed shared endpoint). The route is `ClusterProvider.spec.attribution.auditRoute`, which defaults to the provider's own name. -The operator stores a minimal fact under that route's partition and joins it to a watch event -within a bounded grace window. A missing, late, or absent fact never blocks state capture; it only -changes the author. +The operator appends a minimal fact to that route's per-type fact stream; every process watching the +type follows that stream into a bounded, TTL'd in-process index, and a watch event joins against the +index within a bounded grace window. A missing, late, or absent fact never blocks state capture; it +only changes the author. **Behavior is deterministic and proven by tests.** Given the same observed Kubernetes state, configuration, and Git base, the operator makes the same materialization decisions. Ordering, attribution fallbacks, @@ -379,7 +382,8 @@ flowchart LR RESOLVE["Resolver\nbounded grace window"] GTES[GitTargetEventStream] AFACTS["Audit fact extractor"] - AINDEX[("Redis attribution index\nTTL, keyed for join")] + ASTREAM[("Per-type fact stream\nRedis Streams or in-process ring")] + AINDEX[("In-process fact index\nbounded, TTL, keyed for join")] end subgraph SOURCE["Source cluster selected by ClusterProvider\n(the control cluster when kubeConfig is omitted)"] @@ -409,7 +413,8 @@ flowchart LR PLAN --> PUSH AUDIT -. configured .-> AFACTS - AFACTS --> AINDEX + AFACTS -->|append| ASTREAM + ASTREAM -->|follow| AINDEX AINDEX -. lookup .-> RESOLVE ``` @@ -438,8 +443,9 @@ Following the ConfigMap edit: the remote moved). Separately, the audit path (only when attribution is enabled): kube-apiserver POSTs audit events to -`/audit-webhook/`; [AuditHandler](../internal/webhook/audit_handler.go) extracts a minimal -attribution fact and writes it to that provider's Redis partition with a short TTL. That index is read +`/audit-webhook/`; [AuditHandler](../internal/webhook/audit_handler.go) extracts the minimal +attribution facts and appends them to that route's per-type fact stream. A follower on the watch side +reads the streams for the types it watches into a bounded, TTL'd in-process index. That index is read only by the resolver in step 3; it never creates or repairs object state. **And if the watch had been lost?** A delete that happened while no watch was running is reconciled on @@ -605,17 +611,35 @@ per-mutation change log. ## Optional attribution - **Handler / fact extractor**: [internal/webhook/audit_handler.go](../internal/webhook/audit_handler.go) -- **Attribution index**: [internal/queue/attribution_index.go](../internal/queue/attribution_index.go) +- **Fact transport (per-type stream)**: [internal/queue/fact_stream.go](../internal/queue/fact_stream.go) +- **In-process fact index**: [internal/queue/fact_index.go](../internal/queue/fact_index.go) - **Resolver (grace window join)**: [internal/watch/author_resolver.go](../internal/watch/author_resolver.go) +- **Design**: [attribution-publish-and-join.md](design/attribution-publish-and-join.md), + [attribution-fact-stream.md](finished/attribution-fact-stream.md) -Attribution runs only when `--author-attribution=true`; Redis is then its required state store. A normal +Attribution runs only when `--author-attribution=true`. A normal source posts audit `EventList` payloads to `/audit-webhook/`, where the route is `ClusterProvider.spec.attribution.auditRoute` and defaults to the provider's own name. The bare `/audit-webhook` endpoint is enabled only with `--author-attribution-audit-route-annotation-key`, for a trusted control plane that puts an audit route in each event. There is no supplementary body endpoint or body joiner, because watch (not audit) carries the object body. The handler applies an intrinsic accept gate (StageResponseComplete, a mutating verb, success, non-dry-run, a changed resourceVersion, and the -`/scale` subresource only), then writes the minimal attribution fact to that route's Redis partition. +`/scale` subresource only), then appends the accepted events' facts to that route's per-type fact +stream: **one append per type per request**, not one per event. + +The two halves never call each other. The receiver publishes and returns; the watch side follows the +streams for the types it watches and keeps their facts in a bounded, TTL'd **in-process index**. They +meet only through the keys a fact was filed under. That split is what lets the audit endpoint answer +fast during a rollout, and what lets a fact published by one replica serve a watch running on another +once the streams are shared (the Redis transport below). + +The transport is selectable, because the index is what does the work and the stream is only how facts +travel: + +| `--author-attribution-transport` | Store | When | +|---|---|---| +| `redis` (default) | Redis Streams, per type, with a retention window a restarting process replays | any install; **required** for more than one replica | +| `memory` | an in-process ring | single replica, no Redis; every fact is lost on restart, by design | | Endpoint | Role | |---|---| @@ -645,7 +669,8 @@ Two engineering choices follow directly from that stance: binding, so multi-source deployments must not share a credential across independently trusted sources. - **Tests pin the behavior.** Because a misattribution is a real harm, the attribution and resolver paths carry unit and e2e tests that prove the concrete cases: strong match, weak/last-key match, deletes whose - audit RV differs from the watch RV, missing/late/expired facts, service-account vs human actor, + audit RV differs from the watch RV, a collection delete joined by uid set and by scope, an aggregated + API whose facts carry only a name, missing/late/expired facts, service-account vs human actor, impersonation, and the explicit unresolved outcome, so these certainty guarantees cannot silently regress. ### Attribution fact shape @@ -662,21 +687,66 @@ The fact is the smallest thing needed to name an author, not an object log: | response object resourceVersion | exact watch-event match | | stage timestamp | recency | -The index writes the fact under several join keys, all prefixed by the `ClusterProvider` name: exact -`(provider, GVR, ns, name, uid, rv)`, then `(provider, GVR, ns, name, uid)` (for deletes whose watch RV -differs from the audit RV), then `(provider, GVR, ns, name, rv)` (when UID is absent). Each key carries -the same short TTL (minutes, not hours); old facts are never needed for correctness because watch owns -state. +A `deletecollection` produces one **collection fact** instead: it names no object, so it keeps the +request's selector and whatever uid set the API server returned, and every removal in its scope joins +against it. + +The index files each fact under the **strongest key it has**, first match wins, within a scope of +`(audit route, group/resource)`: `(uid, rv)` *and* `(uid)` together when it has both, else `(uid)`, +else `(rv)`, else `(namespace, name)`. A fact keeps every field it recovered but is not filed under +weaker keys it could never be read by. A watch event always knows its object's uid, so a uid-keyed +fact is never looked up by name, and a second copy would cost memory on every replica for the whole +TTL and answer nothing. The one branch that files twice is `(uid, rv)` plus `(uid)`: the first serves +creates and updates, the second serves removals, one fact answering two different questions. + +Entries carry a short TTL (minutes, not hours) and are bounded per type and in total; expiry is +checked on read, so an aged-out fact is never joined merely because the sweep has not run. Old facts +are never needed for correctness, because watch owns state. ### The resolver and its grace window -A watch event waits a **bounded grace window** (`--author-attribution-grace`, default `3s`) for a matching fact -to arrive, then ships regardless. On a strong match the actor becomes the author: a human or a service -account alike, always named by its own username (e.g. -`system:serviceaccount:flux-system:kustomize-controller`). A weak, conflicting, missing, or expired fact -produces `unknown (attribution unresolved) ` instead of a -guessed actor. A late fact that arrives after a commit has shipped **never rewrites it**. With attribution -disabled the resolver is absent and every commit is committer-authored. +A watch event waits a **bounded grace window** (`--author-attribution-grace`, default `3s`) for +matching evidence, then ships regardless. It does not poll: it registers a waiter under every key its +query could match, looks once, and then sleeps until either a fact that could answer it is applied or +the grace expires. Registering before the first lookup is what closes the race a fact landing in the +gap would otherwise win. There is no Redis call on this path: the fast case is a map read. + +The resolver takes the strongest evidence available, and the tier it used is the `tier` label on +`attribution_resolutions_total`. Who that evidence named is the separate `actor_kind` label, in the +same `user`/`serviceaccount`/`none` vocabulary `commits_total{author_kind}` uses: + +| Tier | Key | What it asserts | +|---|---|---| +| `exact` | uid + rv | this actor produced this exact version | +| `collection_uid` | uid in a collection fact's set | the API server said this request deleted this object | +| `latest` | uid (the object's own delete fact) | who removed it | +| `name` | namespace + name | the same, for a fact with no uid (an aggregated API's usual shape) | +| `latest` | uid (a write fact) | who last *wrote* it; a fallback for a removal | +| `collection_scope` | namespace + selector + window | a collection request covering it was made | +| `resource_version` | rv (the escape hatch for a fact with no uid) | a fact for this exact version, unidentified | +| `absent` | none | nothing usable arrived in time | + +Two rules carry most of the behavior. **A removal never returns on a write fact without looking +further:** the per-object tiers are last-writer-wins, so for a removal they hold whoever last *edited* +the object, which is not who deleted it; such a match is held as a fallback while the wait continues +for evidence about the deletion itself. And **an exact-capable event may not fall through to the +removal tiers:** a create or update presents the resourceVersion its own write produced, so if the +exact tier misses, the uid pointer may name an older, different author. + +No branch on this path depends on the type. Every decision is made on the verb, on whether the event +is a removal, and on which fields the event happens to carry. That is why an aggregated API needed +no special case to attribute. + +On a resolved tier the actor becomes the author: a human or a service account alike, always named by +its own username (e.g. `system:serviceaccount:flux-system:kustomize-controller`). An `absent` +resolution produces `unknown (attribution unresolved) ` +instead of a guessed actor. A late fact that arrives after a commit has shipped **never rewrites it**. +With attribution disabled the resolver is absent and every commit is committer-authored. + +The wait is head-of-line on its watch shard, so a resolution that sits out the grace delays the events +queued behind it on the same `(GitTarget, GVR, scope)` goroutine. That cost, and what it once broke, +is in [Watch Event Ordering](#watch-event-ordering) and +[attribution-removal-wait-options.md](design/attribution-removal-wait-options.md). The CommitRequest controller does **not** read this audit index. A CommitRequest's submitter is named by the `/validate-operator-types` validating admission webhook instead, captured synchronously at admission, @@ -1166,8 +1236,8 @@ flowchart TD H -->|no| Hj[Skip Redis: WatchCursorStore nil; watches cold-replay on restart] Hi --> Hq{author-attribution?} Hj --> Hq - Hq -->|yes| I[Build attribution index + audit fact extractor + audit HTTP server + resolver] - Hq -->|no| J[Configured-author: no attribution index; audit webhook skipped] + Hq -->|yes| I[Select fact transport; start fact index + follower;
wire audit fact extractor + audit HTTP server + resolver] + Hq -->|no| J[Configured-author: no fact streams or index; audit webhook skipped] I --> K[Setup + register Watch Manager] J --> K K --> L[Register GitProvider + ClusterProvider + GitTarget + CommitRequest controllers] @@ -1179,10 +1249,12 @@ Redis is optional in configured-author mode. When `--redis-addr` is set, the cur Redis readiness gate keeps the pod not-ready until Redis is reachable; watches resume from their last stored resourceVersion after a restart. When `--redis-addr` is empty, the cursor store is skipped and watches cold-replay from scratch on restart instead. The binary's `--author-attribution` flag defaults to -on, which requires a non-empty `--redis-addr`: the attribution index is built on the Redis connection, the -audit HTTP handler is wired with the fact extractor, the watch manager gets the author resolver, and the -audit ingress is added to `/readyz`. The Helm chart deliberately passes `--author-attribution=false` by -default, so a first install runs configured-author with no attribution index or audit webhook and every +on: the fact transport is constructed, the fact index and its follower are started, the audit HTTP +handler is wired with the fact extractor, the watch manager gets the author resolver, and the audit +ingress is added to `/readyz`. Which transport it builds is `--author-attribution-transport`: `redis` +(the default) requires a non-empty `--redis-addr`, while `memory` runs the streams in process and is +refused unless `--replica-count` is 1. The Helm chart deliberately passes `--author-attribution=false` +by default, so a first install runs configured-author with no fact streams or audit webhook and every commit is committer-authored. *** @@ -1191,20 +1263,46 @@ commit is committer-authored. - **Source**: [internal/telemetry/exporter.go](../internal/telemetry/exporter.go) -Metrics are exported over OTLP / the metrics server. The audit-attribution path is instrumented: - -- `gitopsreverser_audit_events_total{outcome,category}`: one terminal outcome per audit event (e.g. - `queued`, `stage`, `read_only_or_unknown_verb`, `failed_request`, `dry_run`, - `unchanged_resource_version`, `non_scale_subresource`, `write_error`); -- `gitopsreverser_audit_eventlists_total` / `_eventlist_events_total` / `_eventlist_duration_seconds` - `{outcome}`: the `/audit-webhook` request boundary; -- `gitopsreverser_target_reconcile_completed_total{gittarget_*}`: per-GitTarget reconcile completions - (read by the restart-reconcile guarantee); -- resync/background-apply failure counters so a silently-recovered fault stays visible. - -Per-watch volume/restart/replay metrics and per-attribution result/wait histograms are designed in the -[metrics observability plan](design/metrics-observability-plan.md) but **not yet emitted**; see -[Operational Boundaries](#operational-boundaries). +Metrics are exported over OTLP / the metrics server. The reader's guide to every live family, with +copy-pasteable PromQL, is [interpreting-metrics.md](interpreting-metrics.md). + +The pipeline is one sentence: watch events arrive and are processed into commits. The coverage +follows those stages: + +- **Audit ingress.** `gitopsreverser_audit_events_total{outcome,category,group,version,resource,verb}` + gives one terminal outcome per audit event (`queued`, `stage`, `read_only_or_unknown_verb`, + `failed_request`, `dry_run`, `unchanged_resource_version`, `non_scale_subresource`, + `no_attribution_fact`, `write_error`), and `gitopsreverser_audit_eventlists_total` / + `_eventlist_events_total` / `_eventlist_duration_seconds` `{outcome}` cover the `/audit-webhook` + request boundary. +- **Attribution join.** + `gitopsreverser_attribution_resolutions_total{tier,actor_kind,group,version,resource}` says which + tier of evidence named the author and who it named, per type; + `_resolution_wait_seconds{tier,event_kind,…}` says how long the grace wait cost, split by write and + removal. Splitting the wait by tier is what turned a head-of-line stall from a mystery into a + measurement, and `event_kind` is what the removal wait (the number the grace is tuned from) is + read through. Match coverage is `tier!="absent"`. +- **Fact pipeline.** `_attribution_facts_total{op}` (`written`/`matched`, not subtractable), + `_attribution_fact_index_entries`, `_attribution_fact_index_evictions_total{reason}`, + `_attribution_fact_stream_gaps_total{stream}` (facts lost for good to a trim; should be zero), + `_attribution_fact_stream_decode_errors_total{transport}` (an entry skipped because it could not be + decoded: the loss path with no other symptom), `_attribution_fact_follower_errors_total{transport}` + with `_attribution_fact_follower_last_success_timestamp_seconds` (a wedged follower degrades + attribution cluster-wide), `_attribution_transport_info{transport}`, and + `_attribution_collection_without_uidset_total{reason}`. +- **Git write and reconcile.** `gitopsreverser_commits_total{provider_*,branch,author_kind}` is the + bottom line: `unresolved` means attribution ran and could not name an actor. Alongside it, + `_branch_worker_queue_depth`, `_objects_written_total`, `_resync_sweep_deletes_total`, + `gitopsreverser_target_reconcile_completed_total{gittarget_*}` (read by the restart-reconcile + guarantee), and resync/background-apply failure counters so a silently-recovered fault stays visible. +- **Discovery and encryption.** The API resource catalog and Secret-encryption families. + +**Watch ingestion itself is not instrumented.** Per-type event volume, restarts and `410` rebuilds, +replay cost, recovery mode, and the delay between an event arriving on a shard and being processed are +all designed in the [metrics observability plan](design/metrics-observability-plan.md) and **not yet +emitted**. The attribution half of that plan (the label taxonomy and the silent loss paths) has +shipped; the migration for the label break is in [UPGRADING.md](UPGRADING.md). +See [Operational Boundaries](#operational-boundaries). *** @@ -1220,7 +1318,13 @@ Current limitations: so short reconnects resume a normal watch from that cursor. Kubernetes does not guarantee replay from an arbitrary resourceVersion, so if the apiserver has expired the cursor (`410 Gone`) recovery falls back to `sendInitialEvents` replay or LIST + mark-and-sweep. -- **Per-watch and per-attribution metrics are not yet emitted** (see [Observability](#observability)). +- **Watch-ingestion metrics are not yet emitted.** The attribution join is instrumented, but per-type + watch volume, restarts, replay cost, recovery mode, and shard queue delay are not, so a stalled or + thrashing watch is visible only in logs and in its downstream effects (see + [Observability](#observability)). +- **The in-process attribution transport is single-replica.** `--author-attribution-transport=memory` + is refused with more than one replica, and it loses every unjoined fact on restart by design; a + multi-replica install must use the Redis transport. - **No pull request creation;** the operator writes directly to branches. - **Audit ingress uses a shared-CA trust boundary.** Named `/audit-webhook/` routes and the annotation-routed shared endpoint both require a client certificate signed by the audit CA, but that @@ -1247,7 +1351,7 @@ Current limitations: | [internal/giteaclient/](../internal/giteaclient/) | Gitea helper client | | [internal/manifestanalyzer/](../internal/manifestanalyzer/) | manifest inventory, acceptance, and resync planning | | [internal/manifestreport/](../internal/manifestreport/) | projection of Kubernetes objects into comparable manifest reports | -| [internal/queue/](../internal/queue/) | Redis attribution index (audit facts keyed for the join) and per-watch resume cursors | +| [internal/queue/](../internal/queue/) | attribution fact streams (Redis or in-process), the in-process fact index and its follower, and per-watch resume cursors | | [internal/reconcile/](../internal/reconcile/) | per-GitTarget event stream (watch event → branch worker) | | [internal/rulestore/](../internal/rulestore/) | compiled rule cache | | [internal/sanitize/](../internal/sanitize/) | Kubernetes object sanitization and stable YAML marshal | @@ -1269,6 +1373,10 @@ Deeper dives live under [docs/design/](design/): - [Watch-first ingestion design record](finished/watch-first-ingestion-architecture.md): historical context for the current watch-only object-state model and optional audit attribution. +- [How attribution works: the publish side and the join side](design/attribution-publish-and-join.md): + the two halves, the tier ladder, and the wait. +- [Attribution facts as a stream, not a keyspace](finished/attribution-fact-stream.md): the shipped + transport seam, the in-process index, and what running without Redis costs. - [Watch event ordering under the attribution grace window](facts/watch-event-ordering-and-attribution-grace.md) - [Reconcile via watchlist mark and sweep](spec/reconcile-via-watchlist-mark-and-sweep.md) - [CommitRequest design](spec/commitrequest-design.md) diff --git a/docs/configuration.md b/docs/configuration.md index dabe7377..5c5aaf4a 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -1030,10 +1030,11 @@ Object state comes from Kubernetes **watch**, not from audit. Audit is an option kube-apiserver posts audit events to a named path, `/audit-webhook/`, where the route is `ClusterProvider.spec.attribution.auditRoute` and defaults to the provider's own name. The operator extracts a minimal attribution fact from each (auditID, user, verb, resourceVersion, GVR, namespace, name, UID, -status, timestamps) into a Redis attribution index keyed for the join. A resolver attaches the commit -author to each watch event by matching a fact (by resourceVersion/UID) within a bounded grace window. -The same Redis connection also stores per-watch resume cursors, so short reconnects can resume a normal -watch from the last processed resourceVersion when the apiserver can still serve that history. +status, timestamps) and appends it to a per-type **fact log**: one append per type per request, not one +per event. The watch side follows the log for the types it is watching, holds the facts in a bounded, +TTL'd in-memory index, and attaches the commit author to each watch event by matching a fact within a +bounded grace window. Redis also stores per-watch resume cursors, so short reconnects can resume a +normal watch from the last processed resourceVersion when the apiserver can still serve that history. Named ingress is currently authenticated to the shared audit CA and gated on the provider name existing; it does **not** yet bind a particular client certificate to that provider. Do not use one shared audit @@ -1079,12 +1080,29 @@ Use an annotation that the producing control plane sets consistently as source m routing metadata only: it keeps the audit fact and the watch event in the same source-cluster partition, so a user from one logical cluster can never be credited for a matching object in another. -Valkey/Redis is **optional in configured-author mode**: when `--redis-addr` is set, watch resume cursors are -stored so restarts pick up where they left off; when left empty, watches cold-replay from scratch on -restart instead. When author attribution is enabled (`--author-attribution=true`), a non-empty -`--redis-addr` is required: attribution facts and resume cursors both use the same connection. The Helm -chart defaults to **configured-author** (`attribution.enabled: false`): the audit webhook is unused and every -mirrored-resource commit is authored by the configured committer. +Valkey/Redis is **optional**: when `--redis-addr` is set, watch resume cursors are stored so restarts +pick up where they left off; when left empty, watches cold-replay from scratch on restart instead. The +Helm chart defaults to **configured-author** (`attribution.enabled: false`): the audit webhook is +unused and every mirrored-resource commit is authored by the configured committer. + +Attribution needs a **fact transport**, which is not the same as needing Redis: + +| `--author-attribution-transport` | What carries the facts | `--redis-addr` | +|---|---|---| +| `redis` (default) | Redis streams, one per (audit route, type) | required | +| `memory` | an in-process ring buffer | may be empty | + +Choose `redis` for anything you would call production. It is the only transport whose facts survive a +restart, and the only one that can reach a second process, so it is what an eventual HA topology needs. +Choose `memory` for a single-pod install where running a Valkey StatefulSet to name commit authors +is out of proportion to the benefit. The cost is worth stating plainly: in-memory facts do not survive a +restart, so events in flight across one lose their author. That is already true of any restart today, +which is what keeps the difference small. + +`memory` is **refused at startup with more than one replica**. The transport only carries facts within +one process, so with two replicas an audit request answered by one pod leaves a watch running on the +other with nothing to join, and every commit through it would be authored `attribution-unresolved` with +nothing saying why. The chart passes `replicaCount` in for this check. ```yaml queue: @@ -1107,7 +1125,39 @@ When attribution is enabled, these flags tune the join: - `--author-attribution-grace` (default `3s`): bounded per-event wait for a matching audit fact before a watch event ships authored by the `attribution-unresolved` sentinel. Note the delivery floor: the apiserver's own `--audit-webhook-batch-max-wait` delays every fact by up to that much, so a grace at or - below it will lose actors systematically. + below it will lose actors systematically. The wait ends the moment the fact arrives, so a generous + grace costs latency only when a fact never comes. + + **A removal is the exception, and it is worth budgeting for.** It waits for evidence about the + DELETION rather than settling for the object's last write, so an object edited by one person and + deleted by another is credited to the deleter rather than to the editor. Sometimes no delete fact + ever arrives, most often because the cluster's **audit policy excludes the type** (which is true of + every type in the recommended policy's runtime-noise list). In that case the + removal spends the whole grace before naming the last writer, which is the same answer it would + have given immediately. Measured across one e2e run, where the grace is 10s, the cost lands on + exactly that case and nowhere else: a removal that finds its delete evidence resolves in about + 70ms, while one that never does averages about 3.1s before falling back. Creates and updates do + not consult the deletion tiers at all. The watch shard is single-threaded, so the wait also delays + the events queued behind it on the same `(GitTarget, type, scope)`. Lowering this flag bounds both + directly. + + **Watching a type your audit policy excludes costs more than attribution.** Every removal on such + a type spends the full grace before shipping as the committer, because the fact it is waiting for + was never recorded. If a watched type's commits are consistently committer-authored, check the + audit policy before anything else: a type in the policy's `level: None` list can never be + attributed, and no operator-side setting changes that. +- `--author-attribution-max-facts-per-type` (default `4096`) and `--author-attribution-max-facts` + (default `65536`): how many facts the in-memory index holds, per type and in total, evicted + oldest-first. Per-type is the fair cap: a burst on one noisy type must not evict every other type's + facts. Evictions are counted on `attribution_fact_index_evictions_total{reason}`. +- `--author-attribution-collection-window` (default `30s`): how long after a `deletecollection` a + removal in its scope may still be credited to it. It only has to cover audit batching plus clock skew, + because the removal is attributed at delete-request time, so finalizers do not stretch it. Raising it + widens the risk of crediting an unrelated delete to the collection's actor. +- `--author-attribution-collection-uid-cap` (default `10000`): how many object UIDs a `deletecollection` + fact carries before the set is dropped and the join falls back to scope matching. The fallback is + already correct, so this only decides how often the precise path is taken; drops are counted on + `attribution_collection_without_uidset_total{reason}`. A matched actor is always named by its own username, humans and service accounts alike (e.g. `system:serviceaccount:flux-system:kustomize-controller`); there is no option to collapse service @@ -1115,8 +1165,13 @@ accounts to the committer. ```yaml attribution: + transport: "redis" ttl: "10m" grace: "3s" + maxFactsPerType: 4096 + maxFacts: 65536 + collectionWindow: "30s" + collectionUIDCap: 10000 ``` ## Quickstart vs hand-managed resources diff --git a/docs/design/attribution-branch-findings.md b/docs/design/attribution-branch-findings.md new file mode 100644 index 00000000..45955ee4 --- /dev/null +++ b/docs/design/attribution-branch-findings.md @@ -0,0 +1,316 @@ +# Findings on the attribution fact switchover + +What this branch's loose ends turned out to be, and which of them are measured rather than reasoned. +Five findings: one lab defect that hid every other measurement, three results about how an audit event +identifies its object that the corpus now carries, and one product race that the e2e suite was +failing to report and that is now root-caused and fixed. + +The three identity results are measurements, captured by the mutation lab against the e2e cluster and +committed to the corpus. The race was measured from the resolver's own histogram against a live +cluster. Everything here is evidence rather than inference, except where it says otherwise. + +## 1. The lab was serving the wrong audit route, so nothing was audited + +The mutation lab registered exactly one audit path, `/audit-webhook`. The e2e cluster's audit +kubeconfig posts to `/audit-webhook/default`. The route is NAMED after the `default` +ClusterProvider, because the bare path is the product's shared, annotation-routed endpoint. Go's +`ServeMux` treats a pattern without a trailing slash as an exact match, so every audit event the API +server sent got a 404. + +The store held 133 admission records and 10 watch records, and zero audit records. Every scenario +that requires an audit event timed out (including the seventeen long-standing ones), which reads +exactly like a broken cluster rather than a one-line routing mismatch. That is the expensive part: +the lab's failure mode is indistinguishable from the environment's. + +Fixed by serving the whole `/audit-webhook/` subtree. Row 15 went from a 91-second timeout to passing +in 6.6 seconds, and every scenario passes. The seventeen previously committed corpus rows re-captured +byte-identical, which is the positive result for a re-capture. + +## 2. An aggregated-API removal carries no uid + +Corpus `flunder/aggregated-api-delete/`. The kube-apiserver proxies the request to the extension +server and never decodes what came back, so the audit event's `objectRef` carries: + +```yaml +objectRef: + apiGroup: wardle.example.com + apiVersion: v1alpha1 + name: fl-del # from the URL path + namespace: + resource: flunders +verb: delete +``` + +No `uid`. No `resourceVersion`. No `responseObject` to recover either from. Meanwhile the watch +`DELETED` for the same object carries the full body, uid included. + +## 3. An aggregated deletecollection returns no response body + +Corpus `flunder/aggregated-api-deletecollection/`. One audit record, name-less, with the selector +visible only in the `requestURI`, and no response body: the collection fact therefore carries no uid +set and the join can only proceed by scope. This is precisely the case the deleted response-body expander +produced nothing at all for, and it is the case `collection_scope` exists to serve. + +The scenario deletes three flunders to make the asymmetry unmistakable: three watch `DELETED` events +and three admission records against one audit record that names none of them. + +## Where each verb falls out + +```mermaid +flowchart TD + A[Audit event on an aggregated type] --> B{objectRef has a name?} + B -->|no: create| C[No fact published
rejected at the name gate] + B -->|yes: update, patch, delete| D{body present to backfill
uid and resourceVersion?} + B -->|deletecollection: name-less by nature| E[Collection fact
selector + namespace] + + D -->|no: proxied, so no body| F{uid or resourceVersion
on the fact?} + D -->|yes: ordinary bodied type| G[uid and rv recovered] + + F -->|neither: name only| H[name tier
namespace + name] + G --> I[exact tier: uid + rv
latest tier: uid] + E --> J[collection_uid if a uid set arrived
collection_scope otherwise] + + C --> K[Committer-authored] + H --> L[Attributed to the actor] + I --> L + J --> L + + style C fill:#7f1d1d,color:#fff + style K fill:#7f1d1d,color:#fff + style H fill:#14532d,color:#fff + style L fill:#14532d,color:#fff +``` + +Every verb but the create now reaches an author. Before the name tier, only the collection delete +did: an update, a patch or a single delete published a fact the index discarded on arrival, because +it could be keyed on nothing. The create still reaches no one, and cannot, because its audit event +never says which object it was about. + +## 4. The missing name changes behavior only where there is no body + +This was worth checking, because "the name is not available yet" describes both a `generateName` +create and an aggregated write, and it would be reasonable to expect them to fail the same way. They +do not, and the reason is the fork in the middle of the diagram. + +`IdentityFromAuditEvent` takes namespace, name and uid from `objectRef`, then backfills whatever is +still missing from the event's body: + +```go +preferred, fallback := bodyPriority(event, op) +backfillIdentityFromBody(&id, preferred) +backfillIdentityFromBody(&id, fallback) +``` + +For a `generateName` create on an ordinary type, `objectRef.name` is empty (the API server assigns +the name), but the policy captures at `RequestResponse`, so the `responseObject` carries the assigned +name and uid and the backfill recovers both. The fact publishes and joins normally. + +For an aggregated write there is no body to backfill from, so nothing is recovered. The same empty +field is fatal in one case and harmless in the other, and the discriminator is the body, not the +name. So: yes, the unavailable name does change behavior, but only where the body cannot cover for +it, and that is exactly the aggregated population. + +This is captured rather than left as reasoning. Corpus `configmap/generate-name-create/` is row 18, +the control for the two aggregated rows: it asserts that the `objectRef` carries no name and that the +response body does, so the recovery is evidence in the tree rather than a claim in a paragraph. Put +the three rows side by side and the discriminator is visible without reading any code. + +## 5. The CommitRequest that missed its window: root-caused + +The e2e spec `finalizes a CommitRequest created with metadata.generateName` failed in CI, in the full +local suite, and in an isolated local run. It was not a flake, and it is now fixed at its cause. + +```mermaid +sequenceDiagram + participant T as e2e spec + participant K as kube-apiserver + participant C as controller + participant W as commit window + + T->>K: create Deployment + Note over T,K: +105 ms + T->>K: create CommitRequest (generateName) + C->>C: author resolved from admission record + loop every 2s for ~8s + C->>W: attach enqueued + W-->>C: no open window + end + C->>T: Ready=True reason=NoWindowInGrace sha="" + Note over C,W: 2 seconds later + C->>W: Opening commit window +``` + +Both runs showed the same two-second miss, so it was not load. Two earlier Deployments against the +same GitTarget opened their windows within a couple of seconds; the third did not open one for about +ten, and the request's grace expired at eight. + +### What it was + +Measured, on the branch, from the resolver's own histogram after one run of the commit-request specs: + +| result | resolutions | total wait | mean | +|---|---|---|---| +| `exact_user` | 6 | 1.06s | 0.18s | +| `weak` | 3 | **20.18s** | **6.73s** | + +(These are the label values of the day. `result` has since become `tier` plus `actor_kind`, so +`exact_user` reads `tier="exact", actor_kind="user"`, and `weak` splits: what was measured here is +the uid tier, now `tier="latest"`. The measurement stands; only the names moved.) + +Three removals spent twenty seconds between them. The e2e cluster runs +`--author-attribution-grace=10s`, so each was waiting out most of a full grace, and `weak` is +precisely the tier that holds a removal matched to a WRITE fact. + +That wait is not free, and this is the part that turns a slow resolution into a stalled pipeline. +`streamLiveTargetWatchEvents` processes one shard's events on a single goroutine, and `attachAuthor` +blocks it: a removal that waits out its grace is **head-of-line blocking** for every later event of +that type. Three of them ahead of a create is ten seconds before the create is even looked at, and +the commit window cannot open until it is. The CommitRequest's own grace expires first, and it +reports `NoWindowInGrace` about a window that had not been allowed to exist yet. + +The two-second miss in the diagram is not a tuning problem between two graces. It is where the +serialized waits happened to land. + +### Why the removals were waiting at all + +A removal holds a per-object write fact as a fallback and keeps waiting for evidence about the +deletion, which is correct and deliberate. The question is why that evidence never arrived, when the +delete IS audited and the audit batch interval is one second. + +Because the delete fact was there, and the lookup could not reach it. + +Whether a delete fact carries a uid depends on what the API server answers the request with, and it +answers differently for different deletes. Both shapes are in the corpus: + +- `configmap/finalizer-delete/audit.delete.yaml`: `responseObject` is the **ConfigMap**, so the uid + is recoverable and the fact is filed under the uid tier, where a removal finds it at once. +- `configmap/owner-ref-cascade/audit.delete.cm-parent.yaml`: `responseObject` is a **`Status`**. + There is no uid in it and none in the `objectRef`, so the fact's only key is its name. A + `kubectl delete deployment` is this shape. + +Before the name tier existed, such a fact was published and dropped as unjoinable, so the removal +had no delete evidence at all and waited out the grace. That is the original bug, and it is as old as +the audit event's shape rather than as old as any commit here. + +Adding the name tier stored the fact but did not make it reachable, which is a defect in that change +rather than in the design. `Lookup` returns as soon as the removal ladder yields anything, and the +uid tier yields the object's last WRITE fact, so the name-keyed DELETE fact below was never +consulted. The caller then held the write fact and waited the full grace for evidence that was +already in the index. + +The fix is an ordering rule, and it is the one the removal path already states elsewhere: a fact +about the DELETION outranks a fact about a write, whichever key each is filed under. `lookupRemoval` +now returns the object's own delete fact when the uid tier has one, then consults the name tier for a +delete fact, and only then falls back to the write fact it was holding. A name-keyed WRITE still does +not jump the queue; only removal verbs do. + +### The same measurement, after + +| result | before: n / total / mean | after: n / total / mean | +|---|---|---| +| `exact_user` | 6 / 1.06s / 0.18s | 6 / 0.74s / 0.12s | +| `name` | none reached | 2 / 0.60s / 0.30s | +| `weak` | 3 / 20.18s / **6.73s** | 2 / 0.28s / **0.14s** | +| **total wait** | **21.24s** | **1.63s** | + +The spec passes, and the histogram says it passes for the diagnosed reason rather than by timing luck. +Two resolutions now land on the `name` tier: those are the delete facts that were being stored and +never read. The removals that still resolve `weak` no longer wait for them, because the lookup +reaches the delete evidence before it settles for a write fact, and the shard is no longer blocked +behind a grace that had nothing to wait for. + +### The assertion that hid it + +The spec asserted `Ready=True` and then a non-empty `status.sha`. But `Ready=True` is also the benign +rejection state: `rejectCommitRequest` sets it deliberately for `NoWindowInGrace`, `WindowMismatch` +and `AlreadyPresent`, so that kstatus reads Current rather than Failed. The Ready assertion therefore +passed on a request that committed nothing, and the spec spent the remaining two minutes re-reading +an empty string before reporting: + +```text +Expected + : +not to be empty +``` + +The reason was sitting in the condition the spec never read. `expectCommitRequestCommitted` now +requires the Ready reason to be `Committed` and gives up as soon as any other terminal outcome +appears, because a terminal outcome is final and re-reading it cannot change the answer. The same +failure now reports in ten seconds, naming `NoWindowInGrace` and its message. + +## The decision: a name tier, built + +A name tier is now built rather than proposed. `AuthorFact.Name` is back on the wire, and the +index files a fact under `(namespace, name)` when it carries neither a uid nor a resourceVersion. +`Lookup` consults that tier last, below the rv-only hatch, and reports `AttributionName`. + +The ordering argument is that a name is the weakest per-object evidence available: it is reused after +a delete and recreate, where a uid never is and an rv identifies one specific write. Ranking it last +costs the stronger tiers nothing, because no fact carrying a uid or an rv is ever filed there, and no +query reaches it until every stronger tier has missed. + +What it fixes is exactly the two rows it can reach: an aggregated update or patch, and an aggregated +single delete. Both used to publish a fact the index then discarded, so they shipped +committer-authored whoever ran them. + +Restoring the field is the part worth being explicit about. It was removed during this work on the +observation that no tier read it. That was true of the code and false of the domain: for a whole +population of writes the name is the only identity the audit event carries, so a fact without it +could not be joined at all. "No code reads it" and "nothing could ever read it" are different claims, +and only the second one justifies deleting a field. + +### What the name tier still does not reach + +**The aggregated create.** Its `objectRef` carries no name, and there is no response body to recover +one from, so `AuthorFactFromEvent` rejects it at the name gate and nothing is published for any tier +to join. This is not a gap in the tier; it is a request the API server logged without ever saying +which object it was about. + +Two options remain open for that population, and they are not exclusive: + +**B. Accept that aggregated creates are unattributable.** Document that per-object attribution does +not apply to them, and let them ship committer-authored. Honest, but it makes the guarantee +type-dependent in a way a user cannot predict from the API surface. + +**C. Shorten the wait for facts that provably are not coming.** An aggregated create can be +recognized as unattributable at publish time rather than after a full grace. This attributes nothing; +it stops paying for evidence that cannot arrive. + +### For the window race + +The measurement answered the question this section used to pose. Neither option was right: the window +was not slow to open and the CommitRequest's grace was not too short. The write's event had not been +processed yet, because three removals ahead of it were sitting out most of a ten-second grace each +(20.2s between them, measured) for evidence the index already held. + +That is fixed at its cause, in the lookup ordering. What remains open is the structural half. + +**The head-of-line block is still there.** Any removal that must wait out its grace (a type the +audit policy excludes, a delete whose fact never arrives) still stalls every later event on its shard +for up to ten seconds. This fix removes the largest population that was hitting it, and +does not change the fact that a blocking resolve on a serial goroutine can do this at all. + +Two directions, and they are the same two the wait-versus-poll record already frames: + +1. **Bound the removal's extra wait separately from the grace.** Once a fallback is in hand, the + fact stream for that scope is demonstrably live; what is outstanding is only whether a delete fact + also lands, which is an audit-batch interval rather than a full grace. Small change, needs a + number chosen with evidence. +2. **Stop blocking the shard.** Resolve attribution off the event loop and reassemble in order. This + is the real answer and the larger one; it also fixes every other cause of a slow resolve. + +Worth noting for whoever picks this up: the e2e default of `--author-attribution-grace=10s` makes the +blocking three times worse than the product default of 3s. That is a test-environment choice +amplifying a product behavior, not a product setting anyone runs. + +### A note on the pre-branch probe + +A run against the merge-base (`7ece7310`) was attempted to establish whether the race was new on this +branch, and it did not produce an answer: its bring-up failed before any spec ran, with the manager +rejecting the API server's audit client certificate (`tls: bad certificate`) so the audit pipeline +never warmed up. That is a worktree/cluster-provisioning problem rather than a product difference: +both trees post to the same named audit route and both serve it. + +The comparison turned out not to be needed. The cause is measured directly on the branch, and the +delete-fact shape that triggers it is a property of the Kubernetes audit event rather than of any +commit here. diff --git a/docs/design/attribution-metrics-proposal.md b/docs/design/attribution-metrics-proposal.md new file mode 100644 index 00000000..1d66cf43 --- /dev/null +++ b/docs/design/attribution-metrics-proposal.md @@ -0,0 +1,314 @@ +# Proposal: the attribution metric surface + +> **design**: the attribution half is **built**. `watch_event_queue_seconds` is not: it moved to +> Phase 2 as a pipeline-wide metric. The Phase 1 surface below is now Phase 1 of +> [`metrics-observability-plan.md`](metrics-observability-plan.md), the canonical plan; this document +> is the reasoning trail behind it. Index: [`../INDEX.md`](../INDEX.md) + +Revised after review. An earlier draft proposed thirteen new metric families at once and four of them +would have produced misleading diagnoses. The corrections are spelled out in +[what the first draft got wrong](#what-the-first-draft-got-wrong), because two of the mistakes are +the kind that stay invisible: a series that is always zero, and a gauge that moves in the right +direction while counting the wrong thing. + +What survives is a smaller first release that covers normal-operation health and the loss paths +nothing can currently see, plus a deferred set with the preconditions each one needs. + +## Why change anything now + +**This release has already broken the `result` label.** `exact_deletecollection_item` is gone with the +`deletecollection` rework, replaced by `collection_uid` and `collection_scope`, and `name` is new. +Anything reading the old value stops matching whatever else happens. + +The usual reason to leave a label alone is that changing it breaks consumers, and that cost is paid +once per break. Since this release breaks `result` already, finishing the job costs the same single +migration; deferring it costs a second one later, on a label that will have been wrong twice. Nothing +consumes these metrics yet, which will not stay true. + +That argument covers the renames. It does not cover new metric families, which is why they are phased. + +## Phase 1: the set built first + +| Change | Kind | Why it is in the first release | +|---|---|---| +| `result` becomes `tier` plus `actor_kind` | label rework | the break is already happening | +| `weak` splits into `latest` and `resource_version` | label rework | same break, and `latest` is the tier the removal path turns on | +| `event_kind` on the wait histogram | new label | the wait behavior differs entirely between writes and removals | +| follower errors and last-success timestamp | new family | a wedged follower is silent today | +| `no_attribution_fact` outcome on `audit_events_total` | new value, existing counter | the population that produces no fact, counted where the decision is made | +| a stream-entry decode-error counter | new family | undecodable entries are dropped and skipped past with no log and no metric | +| `attribution_transport_info` | new family | changes how every other metric here is read | +| the four low-risk renames | rename | free while the surface is moving | + +Everything else waits. The renames are in [the table below](#the-renames). + +### `result` becomes `tier` plus `actor_kind` + +Today `result` has seven values and two of them are one tier seen twice: + +```text +exact_user exact_serviceaccount weak collection_uid collection_scope name absent +``` + +`exact` is the only tier that also encodes who the actor was, so counting exact resolutions means +summing two series, and the actor kind cannot be asked of any other tier. There is no way to learn +how many `name` or `collection_uid` resolutions named a service account. + +`gitopsreverser_commits_total` already carries `author_kind` with `user`, `serviceaccount`, +`committer` and `unresolved`, so two metrics currently disagree about the shape of one distinction. + +| Label | Values | +|---|---| +| `tier` | `exact`, `latest`, `resource_version`, `name`, `collection_uid`, `collection_scope`, `absent` | +| `actor_kind` | `user`, `serviceaccount`, `none` | + +**`weak` splits at the same time.** It currently covers both a `latest` (uid) match and the rv-only +hatch, which are different evidence: the object's own last write against a fact that had a +resourceVersion and no uid. The removal path turns on `latest` specifically, and the measurement that +found the window race had to infer "these were `latest` matches held as fallbacks" from a wait +distribution because the label could not say it. + +### `event_kind` on the wait histogram + +`ExactCapable` splits every query into a write or a removal, and the wait design differs completely +between them: a removal holds a fallback and keeps waiting, a write does not. Today the histogram +cannot distinguish an absent write from an absent removal. Adding `event_kind` = `write` / `removal` +makes the removal wait directly queryable, which is the number anyone tuning the grace needs. + +### `watch_event_queue_seconds`, moved to Phase 2 + +This was proposed here and is **not** part of the attribution release. It measures head-of-line +blocking on a watch shard, which is a property of the whole pipeline rather than of attribution, so +the canonical plan took it as the processing-delay stage of §4.2 and scheduled it with the watch +families. The argument for it stands and is kept here: the failure that broke an e2e spec was not a +slow resolution but the delay a slow resolution imposed on the events queued behind it on the same +single-threaded shard, which the wait histogram cannot see because it times each resolution in +isolation. It is also the pressure signal that makes a separate "resolvers waiting" gauge +unnecessary for now. + +### Follower health + +`attribution_fact_follower_errors_total` plus +`attribution_fact_follower_last_success_timestamp_seconds`. + +When the follower fails, `Run` logs and retries with a backoff, and nothing counts it. A follower +that is flapping, or wedged and retrying forever, degrades attribution to committer-authored across +the board, with a rising unresolved rate as the only symptom and nothing pointing at the cause. + +The timestamp matters more than the counter. A counter says errors are happening; only the timestamp +distinguishes "erroring occasionally while making progress" from "has not read anything in ten +minutes", and only the second is an outage. + +### `no_attribution_fact` on `audit_events_total` + +[`internal/audit/outcome`](../../internal/audit/outcome/outcome.go) is already the single bounded +vocabulary for what ingestion did with one event, with a derived `Category` and an e2e invariant that +gates on `category="error"` being zero. An event that is accepted but yields no attribution fact has +no terminal value there today. + +Adding one, in the `Dropped` category rather than `Error`, counts that population at the point where +the decision is made and where the event's type and verb are still on the label set. That is where +the aggregated-API create shows up: it is rejected before publication, so no fact-side counter can +ever see it. + +### A decode-error counter for stream entries + +This is the gap the "every silent drop gets a counter" principle should have caught first and did +not. Both transports do the same thing with an entry they cannot decode: + +```go +facts, err := factsFromMessage(messages[j]) +if err != nil { + continue +} +``` + +and then advance the cursor past it. No log, no metric, no retry. A malformed or future-schema entry +is discarded and the follower moves on as though it had read it. + +`attribution_fact_stream_decode_errors_total` is the whole fix. It belongs in the first release +because it is the one loss path with no symptom at all: unlike a trim gap it is not detectable after +the fact, and unlike a publish failure the API server does not retry it. + +### `attribution_transport_info{transport}` + +An info gauge, value always 1, with `transport="redis"` or `transport="memory"`. + +It is in the first release despite being a new family because it is interpretive metadata rather than +a signal: the two transports have different failure modes, and the same symptom means different +things under each. A burst of unresolved commits after a restart is expected under the in-memory +transport, which loses every fact on restart by design, and is a bug under Redis. Reading any of the +other metrics without knowing which is in force is reading them without knowing the contract. + +If the first release needs to be smaller still, this is the one to cut. + +## What the first draft got wrong + +Each of these was checked against the code. They are recorded rather than deleted, because the reason +each was wrong is more useful than the corrected proposal. + +### `published - filed` is not delivery loss + +The draft proposed a lifecycle counter whose stages could be subtracted: `published` minus `filed` +for delivery loss, `filed` minus `matched` for facts that went unused. + +The subtraction is invalid. `published` counts every fact appended by the audit receiver, for every +type. `filed` would count only facts arriving on streams **this process follows**, which is a subset +chosen by which watches are running. Replay compounds it: a restart re-reads the retention window and +files the same facts again, so the second number can exceed the first without anything being wrong. + +Two counters over different populations do not subtract. Delivery loss has to be measured where +delivery happens, which is what the follower health signals above do. + +### `unfilable` would be a permanently zero series + +The draft claimed a stage for facts the index can file under no key, and that it would show "every +aggregated-API create". + +It would show neither. The publish gate rejects an event with no resolvable name unless it is a +collection verb, so a fact reaching the index always has a name, a uid, an rv, or is a collection +fact. With the name tier in place every one of those files somewhere. The `default` branch in `file` +is unreachable, and a counter on it would be a flat zero that reads as health. + +The aggregated create is the population the draft was reaching for, and it never becomes a fact at +all: it is rejected before publication. Counting it is what the `no_attribution_fact` outcome above +does, on the ingestion side where the event still exists. + +### The named failure metric does not exist + +The draft said publish failures are already visible as `audit_eventlist_*{outcome="write_error"}`. + +`write_error` is a value on `gitopsreverser_audit_events_total`, which is per event. The +`audit_eventlist_*` families are request-level and carry a different outcome set. An alert written +against the name in the draft would report zero forever, which is the worst failure mode a monitoring +change can have. + +### `facts_filed_total{tier}` conflates two models + +The draft proposed counting facts by the tier they were filed under, to show the publish-side +distribution. + +Tiers are resolution outcomes, and they do not partition facts. A fact with a uid and a +resourceVersion is filed under **both** `exact` and `latest`, which the publish-side documentation in +this repository states explicitly. Counting by tier would double-count the most common fact shape and +skew the distribution toward the tier that matters least. + +The question underneath it stays interesting: what shape are facts arriving in, and how many carry +only a name. That needs a fact-shape taxonomy (`uid_rv`, `uid_only`, `rv_only`, `name_only`, +`collection`), which is a different label with different values. Deferred rather than renamed, +because it needs designing rather than editing. + +### `resolvers_waiting` would count registrations + +The draft proposed a gauge incremented when a resolver registers its waiter keys. + +`Await` registers **before** its first lookup, deliberately, so that a fact arriving in the gap wakes +a waiter already listening. Most resolutions then return from that first lookup without ever +blocking. A gauge incremented at registration therefore counts resolutions in flight rather than +resolvers blocked, and it would read as pressure on a healthy system. + +An earlier revision of this document also claimed the existing `factWaiterRegistry.len()` could be +exported directly. It cannot: `len()` returns the number of candidate KEYS holding a waiter, and one +resolver registers under several, so it over-counts by roughly the tier fan-out. + +If it is built later it has to be incremented around the blocking `select` alone. +`watch_event_queue_seconds` measures the same pressure and is scheduled for Phase 2, so this may +never be needed. + +### `streams_behind` does not mean what the name says + +The draft treated it as a backlog depth and an early-loss indicator. + +`behind` is set when **the last read filled its entry budget**, meaning more was waiting when the +read returned. It is the precondition for trim-gap detection rather than a measure of how far behind +the follower is. A stream one entry behind and a stream a thousand entries behind carry the same +value. + +Named and interpreted as drafted it would invite an alert on a condition that occurs during any +ordinary burst. It needs redefining, or replacing with a real lag measure, before it can carry that +meaning. + +### `fact_index_replay_seconds` cannot show what it was for + +The draft proposed measuring replay to show that a restart warms the index before serving. + +There is no replay-complete boundary to measure. The follower runs continuously, streams are added to +the subscription set as watches start, and no readiness barrier gates serving on the index being +warm. A duration recorded today would measure an arbitrary window rather than the property the metric +was proposed to prove. + +The boundary has to exist first. That is a design change with its own value, and it is the same +question the fact-stream record leaves open about HA handover: whether a replica must warm its index +before starting a watch it has taken over. Build the barrier, then measure it. + +## Deferred, and what has to be true first + +| Deferred | Precondition | +|---|---| +| `fact_index_replay_seconds` | a replay-complete boundary and a readiness barrier exist | +| the four stream-scaling metrics | the followed-stream count is large enough to be in question, and `behind` is redefined as real lag | +| fact-shape distribution | a shape taxonomy distinct from the tier taxonomy | +| `resolvers_waiting` | queue delay proves insufficient, and it is measured around the blocking select | +| `fact_index_expired_total` | wanted when tuning the TTL or the caps; low risk, low urgency | + +The stream-scaling set was designed in an earlier revision of this document and that design stands on +its own merits. What changed is the ordering: it is an investigation suite for a question nobody has +observed a problem with, and building it before the health signals above inverts the priority. The +part worth keeping in view is that a count alone cannot answer whether the stream count is reasonable, +because the same number is fine or fatal depending on what it costs. + +## The renames + +| Now | Proposed | Why | +|---|---|---| +| `attribution_resolutions_total{result}` | `{tier, actor_kind}` | one label, two dimensions | +| `attribution_resolution_wait_seconds{result}` | `{tier, event_kind}` | same, plus the write and removal split | +| `attribution_fact_events_total{op}` | `attribution_facts_total{op}` | "events" already means audit events and watch events | +| `attribution_fact_index_size` | `attribution_fact_index_entries` | a gauge should name what it counts | +| `attribution_collection_degraded_total{reason}` | `attribution_collection_without_uidset_total{reason}` | nothing broke; the precise join was unavailable | + +`attribution_fact_index_evictions_total{reason}` and `attribution_fact_stream_gaps_total{stream}` are +unchanged and stay as they are. + +## Reconciling with the canonical plan + +**Done.** [`metrics-observability-plan.md`](metrics-observability-plan.md) declares itself the single +canonical metrics plan, and its attribution taxonomy and this proposal could not both be right. Its +version had drifted from the code when the fact-stream work landed: + +| It specified | The code has | +|---|---| +| `result` includes `conflict` and `expired` | neither value exists | +| `attribution_fact_events_total{op}` includes `expired_unmatched` and `late` | neither op exists | +| `attribution_fact_index_size` is "facts parked in **Redis**" | the index has been in process memory since the fact-stream work | + +So this was never a choice between two live designs. The canonical plan has now absorbed the Phase 1 +surface below as its attribution stage (§4.4 and §5 there), records the drift above as a correction, +and links back here as the reasoning trail. A taxonomy that lives in two places diverges again, and +the canonical plan is the one people are told to read; this document keeps the argument, not the +inventory. + +Two things the canonical plan took from here and generalized, because they are not attribution's +alone: + +- **`watch_event_queue_seconds`** became the processing-delay stage of the whole pipeline rather than + an attribution metric. It measures head-of-line blocking on a watch shard, which attribution + happens to be the loudest current cause of. +- **"every silent drop gets a counter"** became a numbered principle there. It is the rule this + document broke first and then repaired: the decode-error gap below is what a stated principle would + have caught. + +## What to write down + +An [`UPGRADING.md`](../UPGRADING.md) entry with a table of old label and metric names against new +ones, so a query can be rewritten mechanically. It should state that `result` is gone rather than +only describing what replaces it. + +The break itself needs no defense beyond being written down. **Nothing consumes these metrics yet**: +no dashboard ships, no alert rules ship, and no user has been told to build against these names. The +migration costs one entry today and a consumer migration after the first published dashboard, which +is the whole reason the surface is corrected in the same release that broke `result` anyway. + +Worth fixing while in that file: its current attribution entry is headed +`## Unreleased — … (next minor; …)`, which [`AGENTS.md`](../../AGENTS.md) forbids, because by the +time an upgrade guide is read both halves of that heading are false. diff --git a/docs/design/attribution-publish-and-join.md b/docs/design/attribution-publish-and-join.md new file mode 100644 index 00000000..3dffc348 --- /dev/null +++ b/docs/design/attribution-publish-and-join.md @@ -0,0 +1,416 @@ +# How attribution works: the publish side and the join side + +Attribution has two halves that never call each other. One turns an audit event into a FACT; the +other turns a watch event into an AUTHOR by finding a fact. They meet only in the index, through the +keys a fact was filed under. + +This is the reference for what each half does, exactly. For the measurements that produced these +rules, see [`attribution-branch-findings.md`](attribution-branch-findings.md). + +The one thing to carry into both diagrams: **neither half branches on the type.** Every decision is +made on the VERB of the request and on which fields the event happens to carry. Why that is a +requirement rather than a coincidence is the last section. + +## Part 1: the publish side, audit event to fact + +One audit event in. Zero or one fact out, filed under one to three keys. + +```mermaid +flowchart TD + A[Audit event, ResponseComplete] --> B{objectRef present
with a resource?} + B -->|no| X[No fact] + B -->|yes| C{user resolvable?} + C -->|no| X + C -->|yes| D[identity = objectRef namespace, name, uid
then backfill what is missing from the body] + D --> E{verb is
deletecollection?} + + E -->|yes| F[COLLECTION fact
drop uid, name, rv
keep selector from requestURI
keep uid set from the body, if any] + E -->|no| G{identity has
a name?} + G -->|no| X + G -->|yes| H[OBJECT fact
namespace, name, uid, rv, verb, author] + + F --> K[file under: collection namespace] + H --> L{strongest key it has
FIRST MATCH WINS} + L -->|uid and rv| M[file under: exact uid+rv
AND latest uid] + L -->|uid, no rv| N[file under: latest uid] + L -->|rv, no uid| O[file under: rv] + L -->|name, no uid, no rv| P[file under: namespace+name] + L -->|none of these| X + + style X fill:#7f1d1d,color:#fff + style K fill:#14532d,color:#fff + style M fill:#14532d,color:#fff + style N fill:#14532d,color:#fff + style O fill:#14532d,color:#fff + style P fill:#14532d,color:#fff +``` + +Two gates drop an event entirely: no resource, and no resolvable name on a non-collection verb. A +collection request is exempt from the name gate because it names no object by nature, and that is +the one place the verb changes which gate applies. + +A third gate is the fact's whole reason to exist: **no resolvable user, no fact**. It is the one +field the wire contract requires, and the read side enforces it too. `AuthorFact.UnmarshalJSON` +refuses an entry naming nobody, which lands on +`gitopsreverser_attribution_fact_stream_decode_errors_total` rather than being half-absorbed. So a +fact in the index always names an actor, and the metrics can read attribution coverage off the tier +alone. The event that produced no fact is not lost either: it is counted `no_attribution_fact` on +`audit_events_total`, where its type and verb are still in hand. + +The body backfill is why the name gate is survivable. `objectRef` alone often lacks the name or the +uid; `IdentityFromAuditEvent` fills what is missing from the request or response object, preferring +the request object for a delete and the response object otherwise. What the event carries in its body +therefore decides which keys the fact ends up with, and the type has nothing to do with it. + +### Filing picks one branch + +The fact keeps every field it recovered, but it is FILED under one branch only. `file` is a switch on +the strongest key present, and the first matching case wins: a fact with a uid is not also filed +under its name or its resourceVersion, even though it has them. + +The reason is memory. A watch event always knows its object's uid, so it always asks a uid tier +first, and a uid-keyed fact always answers there. A second copy of that same fact under its name +would never be the one read. Storing it anyway costs the entry on every replica following the type, +for the whole TTL, and again on every restart replay, and buys nothing. + +The one branch that files twice is the uid case, and only when it also has a resourceVersion: `exact` +serves creates and updates, `latest` serves removals, and the two answer different questions about +the same object: one fact serving two tiers. + +So the rule is: keep every field, file under exactly the keys a query could reach you by. + +## Part 2: the join side, watch event to author + +One watch event in. It asks the index for the strongest fact about this object, waiting up to the +grace for one to arrive. + +```mermaid +flowchart TD + A[Watch event] --> B[scope = audit route + group/resource] + B --> C{scope known?} + C -->|no| Z[absent: committer-authored] + C -->|yes| D{uid and rv,
and exact-capable?} + + D -->|match| E[exact] + D -->|no match| F{is this a removal?} + + F -->|yes| G{uid in a collection's
uid set?} + G -->|yes| H[collection_uid] + G -->|no| I{latest uid
is a DELETE fact?} + I -->|yes| J[latest: the object's own delete] + I -->|no, it is a write| K[hold it as a fallback] + K --> L{name tier holds
a DELETE fact?} + L -->|yes| M[name] + L -->|no| N{fallback held?} + N -->|yes| O[latest: last writer] + N -->|no| P{collection covers
this scope + selector?} + P -->|yes| Q[collection_scope] + P -->|no| R + + F -->|no| R{rv-only hatch?} + R -->|match| S[resource_version] + R -->|no match| T{name tier?} + T -->|match| U[name] + T -->|no match| Z + + style Z fill:#7f1d1d,color:#fff + style E fill:#14532d,color:#fff + style H fill:#14532d,color:#fff + style J fill:#14532d,color:#fff + style M fill:#14532d,color:#fff + style O fill:#166534,color:#fff + style Q fill:#166534,color:#fff + style S fill:#166534,color:#fff + style U fill:#14532d,color:#fff +``` + +### The tiers, strongest first + +| Tier | Key | `tier` label | What it asserts | +|---|---|---|---| +| exact | uid + rv | `exact` | this actor produced this exact version | +| collection uid | uid in a collection's set | `collection_uid` | the API server said this request deleted this object | +| latest, delete | uid | `latest` | this object's own delete fact | +| name, delete | namespace + name | `name` | this object's own delete fact, when it has no uid | +| latest, write | uid | `latest` | who last wrote it; a fallback for a removal | +| collection scope | namespace + selector + window | `collection_scope` | a collection request covering it was made | +| rv-only | rv | `resource_version` | a fact with an rv but no uid | +| name | namespace + name | `name` | the only key an aggregated write has | +| absent | none | `absent` | committer-authored | + +Who the evidence named is the separate `actor_kind` label (`user` / `serviceaccount` / `none`), so +every row above can be asked about either kind of actor. + +### Two rules that are easy to miss + +**A removal never returns on a write fact without looking further.** The per-object tiers are +last-writer-wins, so for a removal they hold whoever last EDITED the object, which is not who deleted +it. Such a match is held as a fallback while the search continues, and the caller keeps waiting for +delete evidence until the grace expires. A fact about the deletion, filed under any key, ends the +wait immediately. + +**An exact-capable event may not fall through to the removal tiers.** A create or update presents the +resourceVersion its own write produced. If the exact tier misses, the `latest` pointer may name an +older, different author, so the lookup skips straight to the rv hatch and the name tier. + +## The wait, and what changed about it + +The two halves are racing, and the watch side reliably wins. The API server batches audit deliveries +(`--audit-webhook-batch-max-wait`), while the watch event is streamed, so by the time a watch event +needs an author its fact is usually still inside the batch window. The first lookup is a +near-guaranteed miss. That is the whole reason a grace window exists. + +So the resolver does not ask repeatedly. It arms a signal, looks once, and then sleeps until either a +fact that could match it arrives or the grace runs out. + +```mermaid +sequenceDiagram + participant W as watch shard + participant R as waiter registry + participant I as index + participant F as fact follower + + W->>R: register(waiterKeys): one entry per tier this query could match + Note over W,R: registered BEFORE the read, so a fact
landing in the gap still wakes it + W->>I: Lookup + I-->>W: absent + W->>W: select { waiter.ch | ctx.Done | timer.C } + + F->>I: apply fact, file under its keys + I->>R: wake(keys the fact filled) + R-->>W: signal + W->>I: Lookup again + I-->>W: resolved + Note over W: defer unregister, whatever the outcome +``` + +Registering first is what closes the race the old 150ms poll loop papered over by looking again: a +fact delivered between the register and the read wakes a waiter that is already listening. There is +no Redis call on this path. The fast case is a map read; the waiting case is a channel receive. + +### The Go mechanics, because they carry the guarantees + +The registry is `map[factWaiterKey]map[*factWaiter]struct{}`: candidate key to the set of resolvers +blocked on it. That shape is the fan-out. One resolver registers under SEVERAL keys, one per tier its +event could resolve through, and one applied fact wakes every resolver registered under any of the +keys that fact filled. It is a many-to-many join done through an index rather than a broadcast, so a +fact never touches a resolver it could not have answered. + +Four details do real work: + +- **`chan struct{}` with buffer 1.** The signal carries no payload, because the payload is the index + itself: the woken resolver re-reads it. Buffering one means a signal sent while the resolver is + mid-recheck is still there when it comes back around, so it is not lost. +- **Non-blocking send.** `wake` does `select { case ch <- struct{}{}: default: }`, so the goroutine + applying facts is never slowed by a resolver that has not looked yet, and a second signal on an + already-signaled waiter is dropped. One pending wake-up is enough, because the resolver + re-reads everything rather than consuming a queue of events. It also makes the send safe to do + while holding the registry lock, since it cannot block. +- **`select` on three cases.** The resolver waits on the waiter, `ctx.Done()`, and the grace timer + together, so shutdown and the deadline are not special paths. +- **`defer unregister`.** Registration is undone on every exit, including the ones that return early. + The registry exposes a `len()` purely so a test can assert a resolver left nothing behind. + +The loop around the `select` matters too, because a wake-up is only a hint. The resolver re-runs the +whole lookup, and if what arrived was not good enough (a write fact when it needs delete evidence) +it keeps waiting rather than treating the signal as a result. + +### A removal waits for evidence about the deletion + +A match does not always end the wait. The per-object tiers are last-writer-wins, so the fact present +earliest for a removal is usually the object's last WRITE, which says who edited it and nothing about +who deleted it. Returning on that answered "who deleted this" with "who last edited it" whenever +anyone had touched the object first. + +Such a match is now held as a fallback and the wait continues for evidence about the deletion itself. +Waiting never costs an attribution: the worst case returns exactly what returning early would have +returned, one grace later. + +### What that cost, and the fix + +The wait is not free, and this is the part worth knowing before tuning anything. `attachAuthor` runs +on the watch shard's own goroutine, and a shard processes its events serially, so a removal that +waits out its grace is **head-of-line blocking** for every later event of that type. The commit +window for a subsequent write cannot open until its event is processed. + +That turned into a real failure. Measured on the e2e cluster, which runs a 10s grace, three removals +in one run spent 20.18s between them; a later Deployment create queued behind them, its window opened +about ten seconds late, and a CommitRequest created 105ms after the write reported `NoWindowInGrace` +about a window that had not been allowed to exist yet. + +The cause was that the delete evidence was in the index and unreachable. A delete fact only has a uid +if the API server answered the request with the object; when it answers with a `Status` the fact's +only key is its name. `Lookup` returned as soon as the removal ladder yielded anything, and the uid +tier yielded the last write fact, so the name-keyed delete fact below was never consulted. + +`lookupRemoval` now applies one rule: **a fact about the DELETION outranks a fact about a write, +whichever key each is filed under.** The object's own delete fact answers from the uid tier, then +from the name tier, and only then does the held write fact answer. + +| | before | after | +|---|---|---| +| `weak` | 3 resolutions, 20.18s, mean 6.73s | 2 resolutions, 0.28s, mean 0.14s | +| `name` | never reached | 2 resolutions, 0.60s, mean 0.30s | +| total resolver wait | **21.24s** | **1.63s** | + +Still open: the head-of-line block itself. A removal with no delete fact coming at all (a type the +audit policy excludes) still stalls its shard for a whole grace. This change removes the +largest population that was hitting it; it does not change that a blocking resolve on a serial +goroutine can do this at all. See +[`attribution-removal-wait-options.md`](attribution-removal-wait-options.md). + +## What is observable + +Every metric on this path, and the question each answers. + +| Metric | Labels | Answers | +|---|---|---| +| `gitopsreverser_attribution_resolutions_total` | `tier`, `actor_kind`, `group`, `version`, `resource` | which evidence named the author and who it named, per type | +| `gitopsreverser_attribution_resolution_wait_seconds` | `tier`, `event_kind`, `group`, `version`, `resource` | how long the join waited, by tier and by write/removal | +| `gitopsreverser_attribution_facts_total` | `op` = `written` / `matched` | how much of what is published is ever used | +| `gitopsreverser_attribution_fact_index_entries` | none | entries held across every scope | +| `gitopsreverser_attribution_fact_index_evictions_total` | `reason` = `per_type` / `total` | whether the caps are binding | +| `gitopsreverser_attribution_collection_without_uidset_total` | `reason` = `uid_cap` / `no_uids` | how often the precise collection join was unavailable | +| `gitopsreverser_attribution_fact_stream_gaps_total` | `stream` | facts lost for good to a trim | +| `gitopsreverser_attribution_fact_stream_decode_errors_total` | `transport` | entries skipped because they could not be decoded | +| `gitopsreverser_attribution_fact_follower_errors_total` | `transport` | follower reads that failed and were retried | +| `gitopsreverser_attribution_fact_follower_last_success_timestamp_seconds` | none | whether the follower is reading at all | +| `gitopsreverser_attribution_transport_info` | `transport` | which contract the metrics above are read under | +| `gitopsreverser_commits_total` | …, `author_kind` | what reached Git | + +The wait histogram is the one that earns its keep. Splitting wait time BY TIER is what turned the +window race from a mystery into a measurement: the uid-latest tier at a 6.7s mean against the exact +tier at 0.18s said immediately that removals were sitting out their grace, which no aggregate mean +would have shown. `event_kind` now makes that reading direct rather than inferred: the measurement +that found the race had to deduce "these were latest matches held as fallbacks" from the shape of a +distribution, because no label could say it. + +### What is not visible, and one gap that mattered + +**`written` minus `matched` is not delivery loss, and never was.** `written` counts every fact +appended for every type; `matched` counts only facts joined on streams THIS process follows, and a +restart re-files the whole retention window. Two counters over different populations do not subtract. + +The population that motivated the subtraction (name-only delete facts published and silently +discarded) no longer exists: the name tier files them, so `file` returning no keys is now +unreachable behind the publish gate, and a counter on that branch would be a flat zero that reads as +health. What the loss paths needed was measuring where delivery happens: the stream +decode-error counter and the follower's last-success timestamp, both of which now ship. + +**Head-of-line blocking is not measured.** The wait histogram times each resolution in isolation. It +does not measure the delay a slow resolution imposes on the events queued behind it on the same +shard, which is the thing that broke a spec. Time-in-queue per shard, or the age of an event +when it reaches the branch worker, would name it directly. + +**The publish-side tier distribution is not counted.** How many facts land under a name versus a uid +is only discoverable by reading the index. Given that this ratio is the aggregated-API story, it is +worth a counter. + +## The `exact_user` / `exact_serviceaccount` split was a modeling wart, now **fixed** + +`result` is gone; `tier` and `actor_kind` replace it, `weak` split into `latest` and +`resource_version`, and the wait histogram gained `event_kind`. The reasoning is kept below because +it is the argument for the shape, and the migration is in +[`UPGRADING.md`](../UPGRADING.md#0410--the-attribution-metrics-are-relabelled-and-partly-renamed-breaking-for-queries). + +The inconsistency was real rather than cosmetic. `result` should have named the TIER: `weak`, +`name`, `collection_uid`, `collection_scope`, `absent` all did. `exact` was the only one that also +encoded WHO the actor was, which crammed two orthogonal dimensions into one label. + +Two consequences followed directly: + +- counting exact resolutions meant summing two series, and any new actor kind would have multiplied + them again; +- the actor kind could only be asked of the exact tier. There was no way to ask how many `name`-tier + or `collection_uid` resolutions named a service account, because that dimension did not exist + there. + +The decisive argument was that the codebase already modeled it correctly one metric over: +`gitopsreverser_commits_total` carries `author_kind` as its own label, with `user`, `serviceaccount`, +`committer` and `unresolved` as values. So the two metrics disagreed about the shape of the same +distinction. + +The shipped shape is that one: `tier` names the evidence, `actor_kind` matches `commits_total` and is +available on every tier. It was cheap in code, as predicted: the actor kind is derived from the +author string at read time, so nothing new is stored or plumbed. It was still a **breaking metric +change**, taken deliberately in one release with the other label work rather than as a side effect of +an attribution fix. + +## Why it is split into two halves at all + +The split is not an accident of layering. Three requirements each rule out the obvious alternative of +resolving an author inside the audit receiver, or handing it to the watcher over a channel. + +**The audit endpoint must answer fast, and keep answering during a deploy.** The receiver decodes a +batch, appends one entry per type, and returns. It does no lookup, waits for no watcher, and holds no +per-object state. A retried POST may append the same batch twice and that is safe without any +deduplication work on the hot path, because a fact is keyed data rather than a position in a +sequence: the duplicate carries the same author under the same `(uid, rv)`, `latest` is +last-writer-wins over identical content, and a waiter woken twice resolves to the same name. + +**It has to survive more than one replica.** The API server's audit webhook posts through a Service +to whichever replica answers, while a given object's watch shard lives on whichever replica owns that +`GitTarget`. Those are unrelated choices, so the fact and the watcher that needs it routinely land in +different processes. A per-type stream with independent per-reader cursors is exactly the primitive +for that: the receiving replica appends, every replica watching the type reads, and neither needs to +know about the other. An in-process channel works perfectly on one replica and has to be thrown away +on the second. The alternatives are worse in a more expensive way: sticky audit routing would make +the API server's load balancing this operator's problem. + +**Rollouts are the normal state, not the exception.** A replicated deployment is almost always +mid-rollout, reconnecting, or restarting a pod, and those are precisely the cases where plain publish +and subscribe drops facts silently. A resumable stream replays the retention window instead, so a +process that restarts rebuilds its index rather than starting blind. + +**And the delay is not ours to remove.** The batching parameters belong to the API server. Since the +fact is late by construction, the resolver must wait for something rather than ask repeatedly, which +is why the wait is a signal on an in-process index and not a poll against Redis. The old loop ran to +completion on essentially every attributable event, because the first lookup was a near-guaranteed +miss. + +What the split does NOT solve is worth stating too: it stops attribution being an HA blocker, but the +real HA problem is ownership: which replica owns a `GitTarget`, and keeping commits to one +`(GitProvider, branch)` serialized through a single writer. That lives in +[`ha-gittarget-distribution-plan.md`](../future/ha-gittarget-distribution-plan.md). + +## No branch anywhere depends on the type + +Verified across the whole path: `internal/queue`, `internal/auditutil`, and the resolver contain no +comparison against a group, a resource, a kind or an API version. The only `Resource ==` in the path +is `Resource == ""`, a presence check. + +Everything dispatches on one of three things: + +- **the verb**: `deletecollection` publishes a collection fact, and `delete` plus `deletecollection` + are what `isRemovalVerb` recognizes as evidence about a deletion; +- **the operation kind**: `ExactCapable` is false for a removal, which is what unlocks the weaker + tiers; +- **which fields are present**: uid, resourceVersion, name, request body, response body. + +The type appears exactly once, in `factScope{route, groupResource}`, and there it is a PARTITION +rather than a decision: it keeps one cluster's and one type's facts from being handed to another's. +No code reads it to choose a behavior. + +### This is a requirement, not an accident + +A type-based rule would be wrong, and the corpus already proves it. Two ConfigMap deletes, same +cluster, same type, same verb, different shapes: + +- `configmap/finalizer-delete/audit.delete.yaml`: the response object is the **ConfigMap**, so the + uid is recoverable and the fact lands on the uid tiers. +- `configmap/owner-ref-cascade/audit.delete.cm-parent.yaml`: the response object is a **`Status`**, + so there is no uid anywhere and the fact's only key is its name. + +A rule of the form "ConfigMaps behave like this" cannot express that, because the difference is not a +property of ConfigMaps. It is a property of the individual request, decided by propagation policy and +by what the API server chose to return. + +The same principle is what makes aggregated APIs work without ever being mentioned in the code. A +flunder is not special-cased anywhere; it produces events with no uid, and the shape-driven rules +route it to the name tier on their own. When the wardle API was added to the lab, no +attribution code changed to accommodate it. + +So the answer to "is a type check needed?" is no, and adding one would be a regression: it would +replace a rule that reads what the event contains with a guess about what a type usually contains. +The exception that would justify one has not appeared, and the two ConfigMap rows above are the +standing argument that it would be unsound if it did. diff --git a/docs/design/attribution-removal-wait-options.md b/docs/design/attribution-removal-wait-options.md new file mode 100644 index 00000000..5703e059 --- /dev/null +++ b/docs/design/attribution-removal-wait-options.md @@ -0,0 +1,246 @@ +# When a removal should stop waiting for its author + +> **design**: open question, needs a decision. The behaviour it describes is BUILT and shipped in +> the [attribution fact stream](../finished/attribution-fact-stream.md); what is open is whether the +> wait it introduced should be bounded more tightly, and how. Index: [`../INDEX.md`](../INDEX.md) +> +> The short version: a removal now waits for evidence about the DELETION rather than accepting the +> object's last write, which fixed a real mis-attribution. The cost is concentrated in one case — +> a removal for which no delete fact will ever arrive — and there it spends the whole grace window +> to return the answer it would have returned immediately. + +## The question + +Attributing a removal to whoever last EDITED the object names an innocent person as the author of a +deletion they did not perform. Not attributing it at all is honest but useless. Waiting for the +right fact is correct and costs commit latency, on a shard that is single-threaded, so the wait +delays whatever is queued behind it. + +The wait is already bounded by `--author-attribution-grace` (3s by default, 10s in the e2e suite). +The question is whether a removal should be able to stop sooner than that, and on what evidence. + +## First, what actually happens today + +It is worth being precise, because the obvious mental model — try the exact key, then the latest +key, then the collection tiers, each with its own wait — is not what the code does and would be +worse if it did. + +There is **one** wait, and **every** tier is evaluated on **every** wake: + +1. The resolver registers a waiter for all of this event's candidate keys, then reads the index. +2. Any fact applied for any of those keys wakes it. +3. On each wake it re-reads the index and evaluates the whole tier table in one pass, strongest + first. +4. It returns as soon as the answer is one it is willing to keep, and otherwise keeps waiting. + +So the tiers ARE checked in parallel, in the only sense that matters: no tier's wait blocks +another's, and the first satisfying answer wins whichever tier it comes from. The ordering is a +preference among facts that are simultaneously present, not a sequence of attempts. + +What changed with the removal-wait fix is step 4, and only for removals: a match on a per-object +tier whose fact is a WRITE no longer counts as satisfying. It is held as a fallback, and the wait +continues. + +## The situations + +`E` is the event being attributed. "Evidence" means a fact about the deletion: the object's own +delete fact, or a collection fact covering it. + +| # | Situation | Today | Cost | +|---|---|---|---| +| 1 | Removal; the object's own delete fact is already in the index | returns at once, `weak` | none | +| 2 | Removal; a collection fact naming its uid is present | returns at once, `collection_uid` | none — measured at ~70ms | +| 3 | Removal; a collection fact covers its scope | returns at once, `collection_scope` | none | +| 4 | Removal; only a stale WRITE fact is present, and the delete fact arrives during the grace | waits, then returns the deleter | the audit delivery lag, ~1s. **This is the case the fix exists for** | +| 5 | Removal; only a stale WRITE fact is present, and no delete fact ever arrives | waits the FULL grace, then returns the last writer | the whole grace, for an answer available at t=0. **This is the entire cost** | +| 6 | Removal; nothing in the index at all | waits the full grace, then `absent` | unchanged by the fix — this is what the grace has always done | +| 7 | Create or update; its exact fact is present or arrives | returns at once or on arrival | unchanged | +| 8 | Create or update; no fact ever arrives | waits the full grace, then `absent` | unchanged. Aggregated-API writes live here: the audit event carries no uid and no resourceVersion, so nothing joinable is ever published | + +Rows 6 and 8 are worth separating from the argument. They are the pre-existing behaviour of the +grace window, they were not introduced by the removal wait, and no option below removes them. + +**Situation 5 is the whole problem.** Everything else either costs nothing or is unchanged. + +### What situation 5 actually is + +A removal for which no usable audit event reaches us. The dominant cause is not Kubernetes, it is +**the cluster's audit policy** — and that is a much better position to be in, because a policy is a +file someone wrote on purpose. + +- **A type the audit policy drops.** This repository's own recommended policy + ([`policy.yaml`](../../test/e2e/cluster/audit/policy.yaml)) drops `events`, `endpoints`, `nodes`, + `pods`, `bindings`, `componentstatuses` and every `*/status` as runtime noise. Watch any of those + and EVERY removal is situation 5, permanently. +- **An audit route nothing posts to** — the resolver already warns about this one separately. +- A delete whose audit event the **accept gate** drops: a failed request, a dry run. +- A **status-only** change that renders as a removal. + +> **A correction, because this document said otherwise and so did two others.** A graceful pod +> delete was cited as producing "no audit event at all". It does not: the `DELETE` request is +> audited like any other, and under the deletion-as-intent rule that request is precisely the fact +> the join wants. Pods are missing from OUR facts because the policy above drops them. The +> distinction matters — "Kubernetes cannot tell us" is a wall, while "the policy did not ask" is a +> setting, and a knowable one. + +None of these can be distinguished from situation 4 *by looking at the object*. That is the +difficulty: at the moment of resolution, "the fact is late" and "the fact is never coming" look +identical. But they are very distinguishable **by looking at history** — a type whose audit policy +drops it never produces a fact, not once, ever. That is what option F exploits. + +### Aggregated types are a special case, and worse than row 8 suggests + +Row 8 says an aggregated-API write never resolves. Tracing it through the code, the reason is +sharper than "the body is empty", and it changes what could be done about it. + +The kube-apiserver proxies an aggregated request and never sees the object, so the audit event's +`objectRef` carries **no uid and no resourceVersion**, and no name either unless the request URL +supplies one. This is measured, not inferred: corpus `flunder/aggregated-api-write` for the create, +`flunder/aggregated-api-delete` for the single delete, and +`flunder/aggregated-api-deletecollection` for the collection. What each verb then does: + +| Verb on an aggregated type | `objectRef` carries | What happened to the fact | Now | +|---|---|---|---| +| create | nothing: no name, uid or rv | **No fact is published at all.** `AuthorFactFromEvent` rejects it at the "no resolvable name" gate | unchanged, and now COUNTED as `no_attribution_fact` on `audit_events_total` | +| update / patch | the NAME, from the URL path | A fact was published and then **dropped by the index**: with no uid and no rv it could never be joined | **resolves**, on the name tier | +| single delete | the NAME, from the URL path | Same: published, then dropped as unjoinable | **resolves**, on the name tier | +| deletecollection | namespace and selector | **Works.** A collection fact joins by SCOPE, which needs no uid | unchanged, and now two-tiered (`collection_uid` / `collection_scope`) | + +So the observation that creates and updates "measure" while deletes do not was half right, and the +truth was less flattering: for an aggregated type, only the COLLECTION delete was attributable at +all. Everything else either produced no fact or produced one that was discarded on arrival. + +**The tier that fixes two of those rows has shipped.** An update or a single delete carries the +object's NAME, and the watch event carries name and namespace too, so a +`(route, group/resource, namespace, name)` tier joins them. It is weaker than uid — a name is reused +after a delete-and-recreate, where a uid is not — so it sits below every other per-object tier, with +the same care the scope tier gets. It is `tier="name"` on `attribution_resolutions_total`, and +`fact_index.go` documents where it ranks and why. + +Worth recording honestly: the fact's `name` field was REMOVED from the wire during this work, on the +correct observation that no tier read it. That was true, and it is exactly the field this tier +needed back — it was restored to build it. "No code reads it" and "nothing could ever read it" are +different claims, and only the second one justifies deleting a field. + +## The options + +### A. Keep the full grace (status quo) + +Every removal waits up to `--author-attribution-grace` for delete evidence. + +- **For**: simplest; one knob; matches the design's stated principle of waiting before shipping + rather than rewriting after. An operator who cares lowers the grace, which bounds this and every + other wait together. +- **Against**: situation 5 pays the maximum for nothing, and it is not rare — a cluster with pod + churn generates it continuously. The grace is also the wrong lever: lowering it to cut situation 5 + equally cuts situation 4, which is the case worth waiting for. + +### B. A separate, shorter deletion-evidence window + +Wait for delete evidence only up to `D`, then take the fallback; `D` defaults to something like the +audit delivery floor rather than the grace. + +- **For**: recovers most of the latency; keeps situation 4 whenever the batch is on time; small, + local change. +- **Against**: another flag whose right value is a property of the *cluster's* audit configuration + (`--audit-webhook-batch-max-wait`), which this operator cannot see. Set too low it silently + reintroduces the mis-attribution; set too high it buys nothing. It converts a correctness + property into a tuning exercise, which is how the original bug will come back. + +### C. A per-route watermark: stop when the stream has moved past you + +Track, per audit route, the newest fact the follower has applied. If that is newer than this event's +own time plus a skew allowance, then any audit event covering this removal has already been +delivered — so no delete fact is coming, and waiting cannot help. + +- **For**: no new flag, and it answers the actual question rather than approximating it with a + timeout. It uses data already in hand: stream positions are millisecond timestamps, and the index + already applies entries in order. Situation 5 collapses from "the whole grace" to "as soon as the + next fact on that route arrives", while situation 4 is untouched, because the watermark cannot pass + the event until its batch has been processed. +- **Against**: it needs the watermark to be per ROUTE rather than per type, because a quiet type's + stream never advances and would never release the wait. Route-level is the right granularity — + one audit POST carries many types — but it means a route with no traffic at all still waits the + full grace, which is correct but worth stating. Also needs care with clock skew between the API + server's `stageTimestamp` and the operator's clock. + +### D. Wait only when the object was deleted with intent + +The deletion-as-intent rule already tells us a `deletionTimestamp` was set, which means a delete +REQUEST happened, which means an audit event should exist. + +- **For**: targets the wait at removals that provably came from a request. +- **Against**: it does not separate situation 4 from 5 at all. A delete of an audit-excluded type + also comes from a delete request; the request simply is not recorded. The signal is about the + object, and the question is about the audit pipeline. + +### E. Revert the wait; keep only the tier reordering + +Removals return on the first match again. Collection deletes whose response body carried uids are +still credited correctly, because that tier now outranks the write fact and both are present at once. + +- **For**: no latency cost at all; keeps the measured `collection_uid` win. +- **Against**: puts back the mis-attribution for every removal whose delete fact is merely late — + which, given audit batching, is the common case rather than the corner. It trades a correctness + property for latency in the one place the product's core claim lives. + +### F. A per-(route, type) circuit breaker: stop waiting for a fact that has never once arrived + +The resolver already tracks, per audit route, whether attribution has EVER resolved, and warns when +a route has produced a long run of unresolved events. Extend that to `(route, group/resource)` and +use it to decide the wait: a type that has never once produced a fact on this route is not going to +start, so a removal on it should take its fallback immediately. + +This is the [circuit breaker](attribution-wait-poll-vs-push.md#option-c-circuit-break-a-route-that-has-never-resolved-anything) +the earlier record already proposed, applied to the case that turns out to need it. + +- **For**: it targets situation 5 exactly, and for the dominant cause — a type the audit policy + drops — it is not a heuristic but a fact about the configuration: the type is excluded, so it + never publishes, so the counter never moves. The learning is cheap, the mechanism already half + exists, and the failure mode is safe in the right direction: a type that HAS produced facts keeps + waiting, so situation 4 is untouched. It also makes a real misconfiguration visible — "you are + watching a type your audit policy excludes" is exactly the sort of thing an operator wants told. +- **Against**: it needs a warm-up rule, because "never resolved" is also true of a type that has + simply not been written to yet, and getting that wrong would skip the wait for a type that was + about to work. It is per process, so it relearns on restart. And it does nothing for the + intermittent case, where a type produces facts sometimes — for that, C is still the answer. + +## Recommendation + +**F first, then C.** They are complementary rather than alternatives: F removes the permanent case +(a watched type that is never audited) using a signal that is already almost there, and C removes +the transient case (a fact that would have come by now) using data already in the index. F is the +bigger win, because the audit-policy exclusion makes situation 5 *permanent* for a whole type rather +than occasional, and it also surfaces a misconfiguration nobody currently learns about. + +If only one ships, take F. + +**C, with A as the fallback if C proves fiddly,** was the earlier recommendation here and is now +second: it prices the transient case well, but it does not notice that a type is structurally +unauditable, which is where the cost actually concentrates. C is the only option that answers the question +being asked — "is a fact still coming?" — rather than guessing at it with a timeout, and it removes +the cost from situation 5 without touching situation 4. It also adds no configuration surface, which +matters because option B's knob would have to be set from a value the operator holds in the *API +server's* configuration, not this one's. + +B is the pragmatic compromise and should be considered only if C's watermark turns out to be +unreliable in practice. E is a real option, and the honest way to describe it is: it accepts naming +the wrong person some of the time, in exchange for latency. + +## What is not yet measured + +The cost of situation 5 was measured over a single e2e run: removals that find evidence resolve in +about 70ms, removals that never do average about 3.1s. What that run does NOT establish: + +- **how common situation 5 is in a real cluster.** The e2e suite is not a workload; pod churn, + namespace teardown, and controller-driven deletes would all shift it. +- **the throughput effect.** The shard is single-threaded, so the number that matters to a user is + commit latency under a burst of removals, not the mean wait per event. +- **whether the watermark in option C advances often enough** on a quiet route to be worth having. +- **how often a watched type is one the audit policy excludes.** If that is common, option F is the + whole answer; if it is rare, F is a diagnostic and C is the fix. + +A before-and-after against the previous behaviour was attempted and discarded rather than reported: +the two runs' populations differed by more than the change, so the comparison would have been a +workload difference wearing a causal claim's clothes. Deciding between these options on that number +would have been deciding on noise. diff --git a/docs/design/attribution-wait-poll-vs-push.md b/docs/design/attribution-wait-poll-vs-push.md index b156d903..8a8809ce 100644 --- a/docs/design/attribution-wait-poll-vs-push.md +++ b/docs/design/attribution-wait-poll-vs-push.md @@ -2,7 +2,7 @@ > **design**: superseded. Index: [`../INDEX.md`](../INDEX.md) > -> **The decision was taken in [`attribution-fact-stream.md`](attribution-fact-stream.md)**, which +> **The decision was taken in [`attribution-fact-stream.md`](../finished/attribution-fact-stream.md)**, which > replaces the fact keyspace with a per-type Redis stream and an in-memory index. This record is > kept as the reasoning trail: the six options, what each costs, and the measurements that ruled > most of them out. Read it for why, then read the other one for what. @@ -39,7 +39,7 @@ without it. A watch event carries the object, the operation, and the resource id produces a correct commit whether or not anyone ever names an author. [`RedisStore`](../../internal/queue/redis_store.go#L45) holds the resume cursors and is a hard dependency in every mode; -[`AttributionIndex`](../../internal/queue/attribution_index.go#L112) is built on the same connection +`AttributionIndex` is built on the same connection only when the operator asks for author attribution, and it [knows nothing about cursors](../../internal/queue/redis_store.go#L79). Turning attribution off is expressed by leaving `Manager.AuthorResolver` nil, at which point @@ -50,7 +50,7 @@ is optional in the strong sense, and the cost of not using it is zero. **One fact serves every watcher that needs it.** A fact key is `route:::object::` -([`factKeyExact`](../../internal/queue/attribution_index.go#L454)). Notice what is absent from it: +(`factKeyExact`). Notice what is absent from it: the `GitTarget`, the `WatchRule`, the branch, the folder. The key names the *write that happened in Kubernetes*, and it says nothing about which consumer is interested. So when five `GitTarget`s mirror the same `Deployment` into five repositories, the API server posts one audit event, @@ -126,7 +126,7 @@ Two measurements exist in the repository today, and they agree. **The e2e suite reports the wait distribution on every run.** [`reportAttributionStats`](../../test/e2e/e2e_suite_test.go) queries -`gitopsreverser_attribution_resolutions_total` by `result` and prints the +`gitopsreverser_attribution_resolutions_total` by `tier` and prints the `gitopsreverser_attribution_resolution_wait_seconds` histogram, split into resolved and absent because the two populations answer different questions. It also prints how many resolutions succeeded only because e2e widens the grace past the three-second default, which is a direct @@ -194,7 +194,7 @@ and from real installs. expires. Each iteration is not one Redis round trip. -[`LookupAuthorResolution`](../../internal/queue/attribution_index.go#L382) tries up to three keys: +`LookupAuthorResolution` tries up to three keys: the immutable exact key, the `:last` pointer for removals, and the type-scoped rv-only hatch. So a single event that never resolves costs roughly 20 wakeups and 40 to 60 `GET`s, and an event whose fact lands mid-grace pays an average of 75ms of pure poll-interval latency on top of however long @@ -319,7 +319,7 @@ lab's per-scenario report to size it, which is one concrete reason to build the Turn the wait from a poll into a push, driven by the code that receives the audit events. -1. [`writeFactKeys`](../../internal/queue/attribution_index.go#L201) pipelines a `PUBLISH` of each +1. `writeFactKeys` pipelines a `PUBLISH` of each written key onto a per-route channel alongside its `SET`. Pipelined, so the write side pays no extra round trip. See [where to publish from](#where-to-publish-from) for the better placement. 2. The resolver process holds one long-lived subscription per audit route. Routes are bounded by @@ -379,7 +379,7 @@ late-join, and a late-join is precisely what a TTL'd key in Redis serves. #### Carry the fact, not a pointer to it The step above publishes the *key* and has the woken resolver `GET` it. Publishing the -[`AuthorFact`](../../internal/queue/attribution_index.go#L90) itself is better, and it is barely +`AuthorFact` itself is better, and it is barely more work. - **The woken resolver needs no Redis read at all.** The message carries the author, display name, @@ -403,10 +403,10 @@ count, which is the number to watch if this is built. Two placements, and the receiver is the better one. -[`writeFactKeys`](../../internal/queue/attribution_index.go#L201) knows exactly which keys it wrote, +`writeFactKeys` knows exactly which keys it wrote, which makes it the obvious home for a notify-only publish. But it sees one fact at a time, so a `deletecollection` expanding into N facts -([`storeDeleteCollectionFacts`](../../internal/queue/attribution_index.go#L283)) becomes N publishes. +(`storeDeleteCollectionFacts`) becomes N publishes. The receiver sees the whole batch. One audit POST carries an `EventList` of up to `--audit-webhook-batch-max-size` events, decoded once in @@ -417,7 +417,7 @@ publish instead of 400. The batching that causes the delay is the same batching notification cheap. The constraint on that placement is that the receiver must publish only what -[`RecordFact`](../../internal/queue/attribution_index.go#L132) stored. That function drops +`RecordFact` stored. That function drops events with no `objectRef`, no resolvable name, or no user, and expands a `deletecollection` into per-item facts. Publishing the raw `EventList` would notify waiters about facts that do not exist and cannot name anyone. So `RecordFact` needs to return what it wrote, and the receiver accumulates diff --git a/docs/design/metrics-observability-plan.md b/docs/design/metrics-observability-plan.md index 2c9e9a38..2962c7f8 100644 --- a/docs/design/metrics-observability-plan.md +++ b/docs/design/metrics-observability-plan.md @@ -1,36 +1,64 @@ # Metrics & Audit Observability — improvement plan -> **design** — open, not yet built. Index: [`../INDEX.md`](../INDEX.md) +> **design** — partly built. Index: [`../INDEX.md`](../INDEX.md) > -> Status: PROPOSAL — 2026-06-26. **Architecture-led**: [architecture.md](../architecture.md) is the -> spine; every metric below maps to a stage in -> [Common Flows](../architecture.md#common-flows). The live baseline and the +> Status: PLAN — revised 2026-07-29, reconciled to the code after the attribution fact-stream +> switchover. **Architecture-led**: [architecture.md](../architecture.md) is the spine; every metric +> below maps to a stage in [Common Flows](../architecture.md#common-flows). The live baseline and the > documentation bar come from [interpreting-metrics.md](../interpreting-metrics.md). This doc is the -> single canonical metrics plan — it supersedes the per-feature metric notes now in `finished/`. +> single canonical metrics plan — it supersedes the per-feature metric notes now in `finished/`, and +> it now **absorbs** the attribution surface designed in +> [attribution-metrics-proposal.md](attribution-metrics-proposal.md), which stays as the reasoning +> trail for why that surface has the shape it does. ## 1. Why now -After the June 2026 cleanup the live metric surface is honest but **lopsided**: it covers the *edges* -of the pipeline (Git write, discovery catalog, Secret encryption) and is **dark in the middle** — -the two stages that actually carry the product: - -- **Watch ingestion** is the source of object state ([architecture.md → State Ingestion](../architecture.md#state-ingestion-and-not-losing-deletes)) - and has **no direct metrics at all**. -- **Attribution** — naming the author from audit — is the part you most want to watch, and today you - can only see audit *arriving* (`audit_events_total`), not whether it actually *attributes a commit*. - -The goal: make the whole watch-first pipeline observable end-to-end, with **first-class audit / -attribution visibility**, backed by a reference dashboard and a small set of alerts — so there are -serious, honest metrics to show, and the audit subsystem is glass-box. +The product is one sentence: **watch events arrive, and they are processed into commits.** Everything +worth measuring is either a stage of that pipeline, or a way an event can be lost, delayed, or +attributed to the wrong actor on the way through. + +The June 2026 cleanup left a surface that was honest but **lopsided**: it covered the *edges* (Git +write, discovery catalog, Secret encryption) and was dark in the middle. Since then the attribution +stage has been rebuilt and instrumented — the fact keyspace became a per-type fact **stream** that +the watch side follows into an in-process index +([attribution-fact-stream.md](../finished/attribution-fact-stream.md)) — so the middle is no longer +uniformly dark. What is left is a sharper, smaller list: + +- **Watch ingestion** is the source of object state + ([architecture.md → State Ingestion](../architecture.md#state-ingestion-and-not-losing-deletes)) + and still has **no direct metrics at all**. It is now the biggest hole by a wide margin. +- **The delay between an event arriving and being processed** is unmeasured, and it is a proven + failure mode rather than a theoretical one: a slow resolution head-of-line blocks its shard, which + is what broke a CommitRequest e2e spec (see + [attribution-publish-and-join.md → what that cost](attribution-publish-and-join.md#what-that-cost-and-the-fix)). +- **Attribution was instrumented but mislabelled**, and Phase 1 has now fixed it: `result` crammed a + tier and an actor kind into one label, `weak` covered two different kinds of evidence, and the wait + histogram could not tell a write from a removal — exactly the distinction the removal-wait design + turns on. The attribution loss paths that were silent (an undecodable stream entry, a wedged + follower, an accepted event that produces no fact) are counted too. ### The questions the metrics must answer - Is the operator turning cluster changes into commits right now? (liveness) -- **Is audit arriving, is it good, and is it actually putting real names on commits?** (the audit lens) -- When attribution *doesn't* land, why — no audit, weak match, expired fact, no Redis? (degradation) -- Are watches healthy, or thrashing on `410 Gone` / replays? (ingestion health) +- Are watch events arriving, and are they being processed promptly, or queueing? (ingestion health) +- Is audit arriving, is it good, and is it actually putting real names on commits? (the audit lens) +- When attribution *doesn't* land, why — no audit, weak evidence, an unfollowed type, a lost fact? + (degradation) - Is any object state being lost or stalled? (correctness / backpressure) +### Breaking changes are cheap right now, and will not stay cheap + +**Nothing consumes these metrics yet.** No dashboard ships, no alert rules ship, and no user has been +told to build against the current label names. The cost of renaming a label today is one +[`UPGRADING.md`](../UPGRADING.md) entry; the cost after a release with a published dashboard is a +migration for every consumer. This release has already broken the `result` label anyway — the +`deletecollection` rework removed `exact_deletecollection_item` and added `collection_uid`, +`collection_scope`, and `name` — so finishing the job costs the same single migration, and deferring +it costs a second one later on a label that will have been wrong twice. + +So this plan takes the label breaks now, deliberately, and writes them down. It does not take them +quietly. + ## 2. Principles 1. **Architecture is the spine.** Every metric maps to a named stage of @@ -41,40 +69,58 @@ serious, honest metrics to show, and the audit subsystem is glass-box. quality*; the invariant "a missing/late fact changes the author, never the state" ([architecture.md → Optional Attribution](../architecture.md#optional-attribution)) must be *visible*, not just asserted. -4. **Every metric has a recording site and an interpretation.** The thing we just deleted — +4. **Every metric has a recording site and an interpretation.** The thing we deleted in June — defined-but-never-recorded instruments — must never come back. A metric ships with its [interpreting-metrics.md](../interpreting-metrics.md) row (what it measures, one query, what a bad value looks like) in the *same* change. -5. **Label discipline.** Bounded cardinality only: `group`/`version`/`resource` (tens of +5. **Every silent drop gets a counter.** If the pipeline discards, skips, or ages out an event or a + fact, that population is counted where the decision is made. A path that loses data with no + symptom is the failure mode this plan exists to remove. +6. **Label discipline.** Bounded cardinality only: `group`/`version`/`resource` (tens of claimed-and-followable types, not thousands), `verb` (~5), bounded `scope` - (`namespace`/`cluster`), and frozen enum labels (`outcome`, `result`, `reason`). **Never** put an - object's `name`/`namespace` in a label. Identity labels stay prefixed (`provider_*`, `gittarget_*`) - to survive a `honor_labels=false` pod scrape — see the note in + (`namespace`/`cluster`), and frozen enum labels (`outcome`, `tier`, `actor_kind`, `reason`). + **Never** put an object's `name`/`namespace` in a label. Identity labels stay prefixed + (`provider_*`, `gittarget_*`) to survive a `honor_labels=false` pod scrape — see the note in [exporter.go](../../internal/telemetry/exporter.go). -6. **Degradation is loud.** Unresolved attribution, absent Redis, `410` rebuilds, LIST fallback — each - has a metric, so running in a degraded shape is a visible state, not a silent one. +7. **Degradation is loud.** Unresolved attribution, a wedged fact follower, `410` rebuilds, LIST + fallback — each has a metric, so running in a degraded shape is a visible state, not a silent one. +8. **A metric answers a question a reader can act on.** A number nobody can act on is a number that + gets alerted on wrongly; `streams_behind` in §5.4 is the worked example. ## 3. Current state (the map) +The live instrument set is [`exporter.go`](../../internal/telemetry/exporter.go); the reader's guide +to it is [interpreting-metrics.md](../interpreting-metrics.md). + | Pipeline stage (architecture.md) | Live metrics today | Coverage | |---|---|---| | Discovery & catalog | `api_catalog_resources`, `_group_versions`, `_refresh_total`, `_refresh_duration_seconds`, `_generation` | ✅ good | | **Watch ingestion** | — | ❌ **none** | +| **Shard queue / processing delay** | — | ❌ **none** | | **Relevance filter** | — | ❌ **none** | -| **Attribution / audit** | `audit_eventlists_total`, `_eventlist_events_total`, `_eventlist_duration_seconds`, `audit_events_total{outcome,category,group,version,resource,verb}`, `attribution_resolutions_total`, `attribution_resolution_wait_seconds`, `attribution_fact_events_total`, `attribution_fact_index_size` | ✅ coverage and fact-index visibility | -| Git write | `commits_total{provider_*,branch,author_kind}`, `git_operations_total`, `objects_written_total`, `branch_worker_queue_depth`, `resync_sweep_deletes_total` | 🟡 no push latency / conflict | +| Audit ingress | `audit_eventlists_total`, `_eventlist_events_total`, `_eventlist_duration_seconds`, `audit_events_total{outcome,category,group,version,resource,verb}` | ✅ good | +| Attribution publish & join | `attribution_resolutions_total{tier,actor_kind,…}`, `attribution_resolution_wait_seconds{tier,event_kind,…}`, `attribution_facts_total{op}`, `attribution_fact_index_entries`, `_index_evictions_total{reason}`, `_stream_gaps_total{stream}`, `_stream_decode_errors_total{transport}`, `_fact_follower_errors_total{transport}`, `_fact_follower_last_success_timestamp_seconds`, `_collection_without_uidset_total{reason}`, `attribution_transport_info{transport}` | ✅ good (Phase 1 shipped) | +| Git write | `commits_total{provider_*,branch,author_kind}`, `git_operations_total`, `objects_written_total`, `prune_retained_documents_total`, `branch_worker_queue_depth`, `resync_sweep_deletes_total` | 🟡 no push latency / conflict | | Control plane / reconcile | `target_reconcile_completed_total`, `resync_background_failures_total`, `watched_types` | ✅ good | | Secret encryption | `secret_encryption_{attempts,success,failures,cache_hits,marker_skips}_total` | ✅ good | +Two rows changed meaning since the last revision of this plan, and any query written against the +old text is wrong: + +| The old plan said | The code does | +|---|---| +| `result` includes `conflict` and `expired` | neither value ever existed; `result` itself is now gone, replaced by `tier` plus `actor_kind` (§4.4) | +| `attribution_fact_events_total{op}` includes `expired_unmatched` and `late` | `op` is `written` or `matched`, and nothing else | +| `attribution_fact_index_size` is "facts parked in **Redis**" | the index is in **process memory**; Redis (or an in-process ring) carries the fact *stream*, not the index | + ## 4. Target metric model — by pipeline stage -New watch metrics use the shape already sketched in -[watch-first-ingestion-architecture.md → Metrics](../finished/watch-first-ingestion-architecture.md), but this -plan modernizes type labels to the live audit convention: separate `group`, `version`, and -`resource` labels instead of a packed `gvr` string. Keep `version` on watch metrics even though it -adds some series: Git paths and audit metrics already treat version as part of the resource identity, -and a served-version migration should be visible rather than silently folded into the old series. -Attribution metric names are also made more explicit now, before production users depend on them. +New watch metrics use the shape sketched in +[watch-first-ingestion-architecture.md → Metrics](../finished/watch-first-ingestion-architecture.md), +modernized to the live audit convention: separate `group`, `version`, and `resource` labels instead +of a packed `gvr` string. Keep `version` even though it adds series — Git paths and audit metrics +already treat version as part of resource identity, and a served-version migration should be visible +rather than silently folded into the old series. ### 4.1 Watch ingestion (new — the biggest hole) @@ -87,41 +133,60 @@ Attribution metric names are also made more explicit now, before production user | `watch_recovery_total` | counter | `group`, `version`, `resource`, `mode` (`cursor_resume`/`replay`/`list_fallback`) | which recovery path fires — cursor effectiveness vs aggregated-API fallback | | `watch_active` | gauge | `group`, `version`, `resource`, `scope` | open watch goroutines vs claimed set | -Phase 2 adds the bookkeeping needed for `watch_active`; it is not just a recording call. Count -bookmarks at the session receive point before `targetWatchEventResourceVersion` swallows them into -cursor progress. +`watch_active` needs bookkeeping, not just a recording call. Count bookmarks at the session receive +point, before `targetWatchEventResourceVersion` swallows them into cursor progress. -### 4.2 Relevance filter (new) +### 4.2 Processing delay — the head-of-line signal (new) | Metric | Type | Labels | Answers | |---|---|---|---| -| `watch_events_filtered_total` | counter | `group`, `version`, `resource`, `reason` (`sanitized_noop`/`status_only`/`not_followable`/`duplicate`) | is the product-side filter behaving, or masking real changes? A mis-tuned filter is visible, per [watch-first](../finished/watch-first-ingestion-architecture.md). | +| `watch_event_queue_seconds` | histogram | `group`, `version`, `resource` | how long an event waited between arriving on its shard and being picked up | + +This is the failure that broke an e2e spec, and nothing measures it. It was not a slow resolution: it +was the delay a slow resolution imposed on the events queued *behind* it on the same single-threaded +shard. The wait histogram in §4.4 times each resolution in isolation; the ten-second window delay was +only visible by correlating two log lines by hand. -The reason set is the target shape, not a claim that one chokepoint exists today. Phase 3 first -locates or consolidates the scattered filter decisions on the watch-to-Git path, then records the -metric at the smallest honest boundary. +It is also the pressure signal that makes a separate "resolvers currently blocked" gauge unnecessary +for now — see the deferred list in §5.5. -### 4.3 Attribution / audit — **the centerpiece** (§5 expands this) +### 4.3 Relevance filter (new) | Metric | Type | Labels | Answers | |---|---|---|---| -| `audit_events_total` *(have)* | counter | `outcome`, `category`, `group`, `version`, `resource`, `verb` | every audit event by fate (`queued` = attribution fact written) | -| `audit_eventlists_total` / `_eventlist_events_total` / `_eventlist_duration_seconds` *(have)* | counter/hist | `outcome` | the `/audit-webhook` request boundary | -| `attribution_resolutions_total` | counter | `result` (`exact_user`/`exact_serviceaccount`/`weak`/`conflict`/`absent`/`expired`), `group`, `version`, `resource` | **does attribution actually land, per type** | -| `attribution_resolution_wait_seconds` | histogram | `result` | grace-window latency cost (`--author-attribution-grace` tuning) | -| `attribution_fact_events_total` | counter | `op` (`written`/`matched`/`expired_unmatched`/`late`) | fact-index lifecycle — written vs joined vs wasted | -| `attribution_fact_index_size` | gauge | — | facts currently parked in Redis awaiting a join | -| `commits_total` *(have, + intentional label change)* | counter | `provider_*`, `branch`, **`author_kind`** (`user`/`serviceaccount`/`committer`) | **what fraction of commits carry a real name** | +| `watch_events_filtered_total` | counter | `group`, `version`, `resource`, `reason` (`sanitized_noop`/`status_only`/`not_followable`/`duplicate`) | is the product-side filter behaving, or masking real changes? | -### 4.4 Git write (new additions to a covered stage) +The reason set is the target shape, not a claim that one chokepoint exists today. The phase that +builds it first locates or consolidates the scattered filter decisions on the watch-to-Git path, then +records the metric at the smallest honest boundary. + +### 4.4 Attribution / audit — **the centerpiece** (§5 expands this) + +| Metric | Type | Labels | State | +|---|---|---|---| +| `audit_events_total` | counter | `outcome`, `category`, `group`, `version`, `resource`, `verb` | ✅ shipped — `no_attribution_fact` added in the `dropped` category | +| `audit_eventlists_total` / `_eventlist_events_total` / `_eventlist_duration_seconds` | counter/hist | `outcome` | live, unchanged | +| `attribution_resolutions_total` | counter | **`tier`**, **`actor_kind`**, `group`, `version`, `resource` | ✅ shipped — `result` is gone | +| `attribution_resolution_wait_seconds` | histogram | **`tier`**, **`event_kind`**, `group`, `version`, `resource` | ✅ shipped — relabelled and split by write/removal | +| `attribution_facts_total` | counter | `op` (`written`/`matched`) | ✅ shipped — renamed from `attribution_fact_events_total` | +| `attribution_fact_index_entries` | gauge | — | ✅ shipped — renamed from `attribution_fact_index_size` | +| `attribution_fact_index_evictions_total` | counter | `reason` (`per_type`/`total`) | live, unchanged | +| `attribution_fact_stream_gaps_total` | counter | `stream` | live, unchanged | +| `attribution_collection_without_uidset_total` | counter | `reason` (`uid_cap`/`no_uids`) | ✅ shipped — renamed from `attribution_collection_degraded_total` | +| `attribution_fact_stream_decode_errors_total` | counter | `transport` | ✅ shipped — the one loss path that had no symptom at all | +| `attribution_fact_follower_errors_total` | counter | `transport` | ✅ shipped | +| `attribution_fact_follower_last_success_timestamp_seconds` | gauge | — | ✅ shipped — distinguishes "erroring but progressing" from "has read nothing in ten minutes" | +| `attribution_transport_info` | gauge (always 1) | `transport` (`redis`/`memory`) | ✅ shipped — interpretive metadata; changes how every metric above reads | +| `commits_total` | counter | `provider_*`, `branch`, `author_kind` | live, unchanged — the bottom line | + +### 4.5 Git write (new additions to a covered stage) | Metric | Type | Labels | Answers | |---|---|---|---| | `git_push_duration_seconds` | histogram | `provider_*`, `branch` | push latency (re-added with a recording site and doc row) | | `git_push_conflicts_total` | counter | `provider_*`, `branch` | non-fast-forward → fetch/reset/replay retries ([PushAtomic](../../internal/git/git_atomic_push.go) detects a moved remote; [BranchWorker](../../internal/git/branch_worker.go) fetches, rebuilds, and retries) | -| `resync_sweep_deletes_total` | counter | `group`, `version`, `resource` | deletes produced by mark-and-sweep resyncs; steady-state watch deletes use per-event delete-document writes, not sweeps | -### 4.5 Catalog, reconcile, secrets +### 4.6 Catalog, reconcile, secrets Keep as-is (✅ above). One small add: `watch_set_changes_total{gittarget_namespace,gittarget_name,op=open/close}` to see watch churn when @@ -129,98 +194,192 @@ rules/CRDs change (pairs with `target_reconcile_completed_total{trigger=rule_cha ## 5. Deep dive: audit & attribution observability -This is the subsystem you want glass-box. The model -([architecture.md → Optional Attribution](../architecture.md#optional-attribution)): +This is the subsystem to keep glass-box, and it is the one that changed most. The model is now two +halves that never call each other, meeting only through the keys a fact was filed under +([attribution-publish-and-join.md](attribution-publish-and-join.md)): ```text -kube-apiserver --POST--> /audit-webhook --gate--> write attribution fact (Redis, TTL) - | -watch event ---------> resolver waits up to --author-attribution-grace --> join by RV/UID - | - strong match -> user/sa author ; else -> committer +kube-apiserver --POST--> /audit-webhook --gate--> append one entry per type to the fact stream + | (Redis Streams, or an in-process ring) + v + fact follower --> in-process index (bounded, TTL'd) + | +watch event --> resolver registers waiter keys, looks once, --+ + sleeps up to --author-attribution-grace + | + evidence found -> named actor ; else -> unknown (attribution unresolved) ``` -Three lenses, each a dashboard question: +Four lenses, each a dashboard question. -**(a) Is audit arriving and well-formed?** — `audit_eventlists_total{outcome}` (delivery), -`audit_eventlist_duration_seconds` (latency), `audit_events_total{category}` (per-event fate; `error` -must be 0). This is the ingress half we already have. +### 5.1 Is audit arriving and well-formed? -**(b) Is it good enough to attribute?** — `attribution_resolutions_total{result,group,version,resource}` -is the new heart. Phase 1 first changes the `AttributionLookup` / `AuthorResolver` path from a -boolean hit/miss to a structured resolution result, so the metric records facts the code truly knows -instead of inferring them after the fact. It splits every resolved watch event into -`exact_user`, `exact_serviceaccount`, `weak`, `conflict`, `absent`, or `expired` **per type**: +`audit_eventlists_total{outcome}` (delivery), `audit_eventlist_duration_seconds` (latency), +`audit_events_total{category}` (per-event fate; `error` must be 0). This is the ingress half, and it +is already complete. -| Result | Meaning | Work needed | -|---|---|---| -| `exact_user` | exact UID+resourceVersion match for a human user | structured result | -| `exact_serviceaccount` | exact UID+resourceVersion match for a service account, named by its own username | structured result | -| `weak` | non-exact match, such as UID-only or RV-only | define and expose match strength | -| `conflict` | multiple candidate authors share a join key | detect collisions while recording or looking up facts | -| `expired` | a fact existed but aged out before the watch event joined it | add tombstone or last-seen evidence; Redis TTL alone is silent | -| `absent` | no matching fact and no evidence that one expired | structured miss result | - -Match coverage = `(exact_user + exact_serviceaccount) / all` — the share of events that named an actor -(human or service account) rather than falling back to the committer. Real-name coverage is also shown by -`commits_total{author_kind}` below. - -**(c) Are real names actually landing in Git?** — `commits_total{author_kind}`. This is the bottom -line: a wall of `author_kind="committer"` means attribution is effectively off even if audit is -flowing. Pair with `attribution_resolution_wait_seconds` to see whether the grace window is paying -for itself. Adding `author_kind` intentionally changes the existing `commits_total` label contract -before production users depend on it; record it per created commit, so mixed-author batches do not -collapse into a misleading aggregate. - -**Fact-index health** — `attribution_fact_events_total{op}` and `attribution_fact_index_size` show the -Redis side: facts written vs actually matched vs `expired_unmatched` (wasted writes) vs `late` (arrived -after the commit shipped — never rewrites, per the invariant, but worth seeing). A high -`expired_unmatched` rate with low coverage is the signature of an audit/watch RV mismatch. - -**Live investigation** — for "watch what's happening right now," metrics aggregate by design. Keep the -per-outcome labels specific enough that an operator can move from the dashboard to the raw audit/debug -logs without needing a second metrics taxonomy. - -**Adaptive payoff** — once `attribution_resolutions_total{result="absent",group,version,resource}` and -a per-type last-seen signal exist, the resolver can **skip the grace wait for types audit never -covers** (no point delaying a watch event 3s for a fact that never comes). The metric is the -prerequisite; the optimization is a Phase 4 follow-up. +One value was missing from it, and Phase 1 added it. +[`internal/audit/outcome`](../../internal/audit/outcome/outcome.go) is the single bounded vocabulary +for what ingestion did with one event, and an event that is accepted but yields **no attribution +fact** had no terminal value there — it was counted `queued`, which claimed an append that was never +owed. `no_attribution_fact` in the `Dropped` category (not `Error`, so the e2e invariant is intact) +counts that population at the point where the decision is made, while the event's type and verb are +still on the label set. That is where the aggregated-API create shows up: it is rejected before +publication, so no fact-side counter can ever see it. + +### 5.2 Is the evidence good enough to name an actor? + +`attribution_resolutions_total` is the heart, and its label was wrong. + +`result` had seven values and two of them were one tier seen twice: + +```text +exact_user exact_serviceaccount weak collection_uid collection_scope name absent +``` + +`exact` is the only tier that also encodes *who* the actor was, so counting exact resolutions means +summing two series, and the actor kind cannot be asked of any other tier — there is no way to learn +how many `name` or `collection_uid` resolutions named a service account. Meanwhile `commits_total` +already carries `author_kind` with `user`, `serviceaccount`, `committer`, and `unresolved`, so the +two metrics disagreed about the shape of one distinction. + +**Shipped:** + +| Label | Values | +|---|---| +| `tier` | `exact`, `latest`, `resource_version`, `name`, `collection_uid`, `collection_scope`, `absent` | +| `actor_kind` | `user`, `serviceaccount`, `none` | + +`weak` split at the same time. It covered both a `latest` (uid) match and the rv-only +escape hatch, which are different evidence: the object's own last write, against a fact that carried +a resourceVersion and no uid. The removal path turns on `latest` specifically, and the measurement +that found the window race had to *infer* "these were `latest` matches held as fallbacks" from a wait +distribution, because the label could not say it. + +The tier ladder itself, strongest first, is documented in +[attribution-publish-and-join.md → the tiers](attribution-publish-and-join.md#the-tiers-strongest-first); +the operator-facing reading of each value is in +[interpreting-metrics.md](../interpreting-metrics.md#audit-attribution-optional). + +**Match coverage** is the share of resolutions that named an actor rather than producing the explicit +unresolved author. It is `tier != "absent"` — *not* `tier =~ "exact.*"`, which would read the +collection and name tiers as misses. + +### 5.3 How long did it wait, and for what kind of event? + +`attribution_resolution_wait_seconds` carries `event_kind` (`write` / `removal`). `ExactCapable` +splits every query into a write or a removal, and the wait design differs completely between them: a +removal holds a fallback and keeps waiting for evidence about the deletion, a write does not. The +histogram used to put an absent write and an absent removal in one series, and the removal wait is +the number anyone tuning `--author-attribution-grace` actually needs. + +Splitting wait time by outcome is what turned the window race from a mystery into a measurement: +the uid-latest tier at a 6.7 s mean against the exact tier at 0.18 s said immediately that removals +were sitting out their grace, which no aggregate mean would have shown. `event_kind` makes that +reading direct instead of inferred. + +### 5.4 Is the fact pipeline itself healthy? + +The publish side, the transport, and the follower are three places a fact can be lost. All three are +counted now; the last two were added in Phase 1. + +- **Decode errors — the sharpest gap.** Both transports did the same thing with an entry they cannot + decode: `continue`, then advance the cursor past it. No log, no metric, no retry. A malformed or + future-schema entry was discarded and the follower moved on as though it had read it. + `attribution_fact_stream_decode_errors_total` (plus a log line) is the whole fix, and it was first + in line because it is the one loss path with **no symptom at all**: unlike a trim gap it is not + detectable after the fact, and unlike a publish failure the API server does not retry it. +- **Follower health.** When the follower fails, `Run` logs and retries with a backoff. Nothing + counted it. A follower that is flapping, or wedged and retrying forever, degrades attribution to + committer-authored across the board, with a rising unresolved rate as the only symptom and nothing + pointing at the cause. The **timestamp matters more than the counter**: a counter says errors are + happening; only `..._last_success_timestamp_seconds` distinguishes "erroring occasionally while + making progress" from "has not read anything in ten minutes", and only the second is an outage. +- **Transport identity.** `attribution_transport_info{transport}` is an info gauge, value always 1. + It is interpretive metadata rather than a signal: the two transports have different failure modes, + and the same symptom means different things under each. A burst of unresolved commits after a + restart is *expected* under the in-memory transport, which loses every fact on restart by design, + and is a *bug* under Redis. Reading any other metric here without knowing which transport is in + force is reading it without knowing the contract. +- **Already live and worth keeping:** `attribution_fact_stream_gaps_total{stream}` (facts lost for + good to a trim — should be zero), `attribution_fact_index_evictions_total{reason}` (the caps are + binding), and `attribution_collection_without_uidset_total{reason}` (the precise collection join + was unavailable, so the resolution fell to the scope tier). + +One live signal is **not** ready to be exported. `behind` on a followed stream is set when the last +read filled its entry budget, meaning more was waiting when the read returned. It is the precondition +for trim-gap detection, not a measure of how far behind the follower is — a stream one entry behind +and a stream a thousand entries behind carry the same value. Exported as `streams_behind` it would +invite an alert on a condition that occurs during any ordinary burst. It needs redefining as real lag +before it can carry that meaning. + +### 5.5 Deferred, and what has to be true first + +| Deferred | Precondition | +|---|---| +| `fact_index_replay_seconds` | a replay-complete boundary and a readiness barrier exist at all — the follower runs continuously, streams join the subscription set as watches start, and nothing gates serving on a warm index, so a duration recorded today would measure an arbitrary window | +| the stream-scaling set (followed-stream count, per-stream read cost, lag) | the followed-stream count is large enough to be in question, and `behind` is redefined as real lag | +| fact-shape distribution (`uid_rv`/`uid_only`/`rv_only`/`name_only`/`collection`) | a shape taxonomy distinct from the tier taxonomy. Counting facts *by tier* is invalid: a fact with a uid **and** an rv is filed under both `exact` and `latest`, so tiers do not partition facts | +| `resolvers_waiting` | queue delay (§4.2) proves insufficient, **and** it is incremented around the blocking `select` alone — `Await` registers *before* its first lookup on purpose, so a gauge at registration counts resolutions in flight rather than resolvers blocked | +| `fact_index_expired_total` | wanted when tuning the TTL or the caps; low risk, low urgency | + +The full record of what an earlier draft of this surface got wrong, and why each mistake was +invisible, is in [attribution-metrics-proposal.md](attribution-metrics-proposal.md#what-the-first-draft-got-wrong). ## 6. The reference dashboard -Grafana, one dashboard, top-down. The **Audit & Attribution** row is the marquee. PromQL is given so a -panel is copy-pasteable. +Grafana, one dashboard, top-down. The **Audit & Attribution** row is the marquee. PromQL is given so +a panel is copy-pasteable. The attribution queries below are against the shipped Phase 1 labels; the +watch and push families they sit beside are not emitted yet. **Row 0 — SLO header (stat panels):** - Commit rate: `sum(rate(gitopsreverser_commits_total[5m]))` - Audit errors (must be 0): `sum(rate(gitopsreverser_audit_events_total{category="error"}[5m]))` -- Attribution match coverage %: `sum(rate(gitopsreverser_attribution_resolutions_total{result=~"exact_.*"}[5m])) / sum(rate(gitopsreverser_attribution_resolutions_total[5m]))` +- Attribution match coverage %: + `sum(rate(gitopsreverser_attribution_resolutions_total{tier!="absent"}[5m])) / sum(rate(gitopsreverser_attribution_resolutions_total[5m]))` - Push latency p95: `histogram_quantile(0.95, sum by (le)(rate(gitopsreverser_git_push_duration_seconds_bucket[5m])))` - Max worker queue depth: `max(gitopsreverser_branch_worker_queue_depth)` +- Transport in force: `gitopsreverser_attribution_transport_info` (a legend, not a threshold) **Row 1 — AUDIT & ATTRIBUTION (marquee):** - *Live audit stream by type* (timeseries): `sum by (group,version,resource)(rate(gitopsreverser_audit_events_total[1m]))` - *Audit outcome mix* (stacked): `sum by (category,outcome)(rate(gitopsreverser_audit_events_total[5m]))` -- *Attribution match coverage by type* (timeseries): +- *Attribution coverage by type* (timeseries): ```promql - sum by (group,version,resource)(rate(gitopsreverser_attribution_resolutions_total{result=~"exact_.*"}[5m])) + sum by (group,version,resource)(rate(gitopsreverser_attribution_resolutions_total{tier!="absent"}[5m])) / sum by (group,version,resource)(rate(gitopsreverser_attribution_resolutions_total[5m])) ``` +- *Evidence mix* (stacked): `sum by (tier)(rate(gitopsreverser_attribution_resolutions_total[5m]))` — + a shift from `exact` toward `collection_scope` or `name` is a quality regression even while + coverage holds flat. +- *Actor mix* (pie): `sum by (actor_kind)(rate(gitopsreverser_attribution_resolutions_total[15m]))` - *Commit author mix* (pie/stacked): `sum by (author_kind)(rate(gitopsreverser_commits_total[15m]))` -- *Grace-window wait p95 by result* (timeseries) with an `--author-attribution-grace` threshold - line: - `histogram_quantile(0.95, sum by (le,result)(rate(gitopsreverser_attribution_resolution_wait_seconds_bucket[5m])))` -- *Fact-index health* (timeseries): `sum by (op)(rate(gitopsreverser_attribution_fact_events_total[5m]))` + `gitopsreverser_attribution_fact_index_size` +- *Removal wait p95* (timeseries) with an `--author-attribution-grace` threshold line — the panel the + grace window is tuned from: + + ```promql + histogram_quantile(0.95, sum by (le,tier)( + rate(gitopsreverser_attribution_resolution_wait_seconds_bucket{event_kind="removal"}[5m]))) + ``` + +- *Fact pipeline health* (timeseries): `sum by (op)(rate(gitopsreverser_attribution_facts_total[5m]))`, + alongside `gitopsreverser_attribution_fact_index_entries` +- *Fact loss* (timeseries, should be flat zero): + `sum(rate(gitopsreverser_attribution_fact_stream_gaps_total[5m]))`, + `sum(rate(gitopsreverser_attribution_fact_stream_decode_errors_total[5m]))`, + `sum by (reason)(rate(gitopsreverser_attribution_fact_index_evictions_total[5m]))` +- *Follower liveness* (stat): `time() - gitopsreverser_attribution_fact_follower_last_success_timestamp_seconds` - *Top dropped audit outcomes* (table): `topk(10, sum by (resource,verb,outcome)(rate(gitopsreverser_audit_events_total{category="dropped"}[5m])))` -- *EventList ingress + decode errors* (timeseries): `sum by (outcome)(rate(gitopsreverser_audit_eventlists_total[5m]))` **Row 2 — WATCH INGESTION:** - Events/sec by type: `sum by (group,version,resource,type)(rate(gitopsreverser_watch_events_total[5m]))` +- Queue delay p95 — the head-of-line panel: + `histogram_quantile(0.95, sum by (le,group,version,resource)(rate(gitopsreverser_watch_event_queue_seconds_bucket[5m])))` - Restarts / `410` pressure: `sum by (group,version,resource,reason)(rate(gitopsreverser_watch_restarts_total[15m]))` - Replay p95: `histogram_quantile(0.95, sum by (le,group,version,resource)(rate(gitopsreverser_watch_replay_seconds_bucket[5m])))` - Recovery mode mix (cursor vs replay vs list): `sum by (mode)(rate(gitopsreverser_watch_recovery_total[15m]))` @@ -236,75 +395,127 @@ panel is copy-pasteable. - Allowed resources, degraded group/versions (`> 0` red), refresh outcome mix, encryption failure rate. -> The dashboard ships as JSON under `docs/dashboards/` (or the chart) so it is versioned with the code. -> I can generate the Grafana JSON once Phase 1 metrics exist. +> The dashboard ships as JSON under `docs/dashboards/` (or the chart) so it is versioned with the +> code. It is built after the §4.4 relabel lands, so it is never written against a name that is +> already scheduled to change. ## 7. Cardinality & cost - `group`/`version`/`resource` is bounded by **claimed ∩ followable** types (tens), `verb` ~5, - `scope` is a bounded enum (`namespace`/`cluster`), and all other labels are frozen enums - (`result` 7, `outcome` ~10, `category` 4, `author_kind` 3, `mode` 3). Worst case is a few thousand - series total — comfortable for Prometheus. -- **No object identity in labels** (no `name`/`namespace` of watched objects) — that is the only thing - that would blow up cardinality, and it is forbidden by principle 5. + `scope` is a bounded enum (`namespace`/`cluster`), and all other labels are frozen enums (`tier` 7, + `actor_kind` 3, `event_kind` 2, `outcome` ~11, `category` 3, `author_kind` 4, `mode` 3). Worst case + is a few thousand series total — comfortable for Prometheus. +- The one place to watch is `attribution_resolution_wait_seconds`, which is a histogram carrying + `tier` × `event_kind` × the type triple. It already carries the type triple today; `event_kind` + doubles it at most, and removals and writes rarely both occur for every tier. +- **No object identity in labels** (no `name`/`namespace` of watched objects) — that is the only + thing that would blow up cardinality, and principle 6 forbids it. - Histograms reuse shared bucket sets (sub-second→minutes), as today. -- `diag_all` is **opt-in and `MAXLEN`-bounded**; off in prod by default. ## 8. Alerts / SLOs | Alert | Expression (sketch) | Meaning | |---|---|---| -| Audit fact-store errors | `rate(audit_events_total{outcome="write_error"}[10m]) > 0` | Redis fact writes failing | -| Attribution match coverage drop | match coverage `< 0.5` for 30m while audit flowing | facts stopped matching watch events | -| Grace window saturating | `attribution_resolution_wait_seconds{result=~"absent\|expired"}` p95 → `--author-attribution-grace` | misses are waiting the full grace; raise grace or skip never-attributed types | +| Audit fact-store errors | `rate(gitopsreverser_audit_events_total{category="error"}[10m]) > 0` | fact appends are failing — check the transport | +| Fact stream loss | `rate(gitopsreverser_attribution_fact_stream_gaps_total[10m]) > 0` | the stream was trimmed past this process's position; those facts are gone | +| Undecodable fact entries | `rate(gitopsreverser_attribution_fact_stream_decode_errors_total[10m]) > 0` | a schema or version mismatch on the stream; facts are being skipped | +| Fact follower wedged | `(time() - …_fact_follower_last_success_timestamp_seconds > 600) or (…_transport_info == 1 unless on() …_fact_follower_last_success_timestamp_seconds)`, `for: 10m` | attribution is degrading to committer-authored cluster-wide | +| Attribution coverage drop | coverage (`tier!="absent"`) `< 0.5` for 30m while audit is flowing | facts stopped matching watch events | +| Grace window saturating | `attribution_resolution_wait_seconds{tier="absent",event_kind="removal"}` p95 → `--author-attribution-grace` | removals are sitting out the full grace; raise grace, or skip the wait for never-attributed types | +| Shard queue delay | `watch_event_queue_seconds` p95 approaching the grace window | head-of-line blocking; events are queued behind slow resolutions | | Watch restart storm | `rate(watch_restarts_total{reason="410_gone"}[15m])` spike | RV churn / compaction pressure | | List fallback in use | `rate(watch_recovery_total{mode="list_fallback"}[1h]) > 0` | an aggregated API isn't honoring streaming list | | Worker backing up | `branch_worker_queue_depth` rising, not draining | stalled remote | | Degraded API surface | `api_catalog_group_versions{state="degraded"} > 0` | broken APIService | -## 9. Implementation phases +The follower row needs both arms, and the second is the one that is easy to leave out. The gauge is +not emitted until the follower's first successful read, so `time() - ` returns **no series** +for a follower that has been wedged since startup — a transport unreachable at boot, which is +precisely the outage the metric exists for. The `unless` arm fires on the gauge's ABSENCE while +`attribution_transport_info` says a follower is running, and `for: 10m` keeps an ordinary restart's +gap from tripping it. Stamping the gauge at start instead would remove the arm at the cost of +claiming a success that never happened, which is worse: it reads as health for the first ten minutes +of every outage. + +Note the first row's metric: `write_error` is a value on `gitopsreverser_audit_events_total`, which +is **per event**. The `audit_eventlist_*` families are request-level and carry a different outcome +set; an alert written against `audit_eventlist_*{outcome="write_error"}` reports zero forever, which +is the worst failure mode a monitoring change can have. -Each phase ships: recording sites → unit tests (manual-reader assertions) → `interpreting-metrics.md` -rows → dashboard panels → alerts, validated per [AGENTS.md](../../AGENTS.md) (`fmt`→`generate`→ -`manifests`→`vet`→`lint`→`test`→`test-e2e`, e2e sequential). **No metric merges without its doc row.** +## 9. Implementation phases -1. **Attribution (implemented).** Structured resolver result, - `attribution_resolutions_total`, `attribution_resolution_wait_seconds`, - `attribution_fact_events_total`, `attribution_fact_index_size`, the intentional - `commits_total{author_kind}` label-contract change, and `resync_sweep_deletes_total`. Recording sites: +Phases 1-3 each ship: recording sites → unit tests (manual-reader assertions) → +`interpreting-metrics.md` rows, validated per +[AGENTS.md](../../AGENTS.md) (`fmt`→`generate`→`manifests`→`vet`→`lint`→`test`→`test-e2e`, e2e +sequential). **No metric merges without its doc row.** The dashboard JSON and the alert RULES are +Phase 4 for one reason: a panel or an alert written against a family that is still being designed is +a query nobody re-checks once it stops matching. The alert *sketches* in §8 are the specification +those rules are written from, not shipped rules. + +0. **Attribution join — done.** Structured resolver result, `attribution_resolutions_total`, + `attribution_resolution_wait_seconds`, `attribution_fact_events_total`, + `attribution_fact_index_size`, `_index_evictions_total`, `_stream_gaps_total`, + `_collection_degraded_total`, and the `commits_total{author_kind}` label change all ship today. + Sites: [author_resolver.go](../../internal/watch/author_resolver.go), + [fact_index.go](../../internal/queue/fact_index.go), + [author_fact.go](../../internal/queue/author_fact.go), + [branch_worker.go](../../internal/git/branch_worker.go). +1. **Attribution surface correction + the silent loss paths — done.** The §4.4 relabel (`tier` + + `actor_kind`, `weak` → `latest` / `resource_version`, `event_kind` on the wait histogram), the + four renames, `no_attribution_fact` on `audit_events_total`, the stream decode-error counter, the + follower error counter and last-success gauge, and `attribution_transport_info` all ship, each + with its recording site, a manual-reader unit test, and its + [interpreting-metrics.md](../interpreting-metrics.md) row. The label break is written up in + [`UPGRADING.md`](../UPGRADING.md). `AttributionResult` in + [author_fact.go](../../internal/queue/author_fact.go) is the tier vocabulary — the split lives in + the enum, not at the metric boundary, so the resolver names the tier it actually took. Sites: [author_resolver.go](../../internal/watch/author_resolver.go), - [attribution_index.go](../../internal/queue/attribution_index.go), - [branch_worker.go](../../internal/git/branch_worker.go), and - [resync_flush.go](../../internal/git/resync_flush.go). Ship dashboard Row 0 + Row 1. *This alone gives - serious, demoable audit metrics.* -2. **Watch ingestion.** `watch_events_total`, `watch_restarts_total`, `watch_replay_seconds`, - `watch_recovery_total`, `watch_active`. Sites: - [target_watch.go](../../internal/watch/target_watch.go), [manager.go](../../internal/watch/manager.go). - Ship Row 2. + [fact_index.go](../../internal/queue/fact_index.go), + [fact_stream.go](../../internal/queue/fact_stream.go), + [outcome.go](../../internal/audit/outcome/outcome.go), + [audit_handler.go](../../internal/webhook/audit_handler.go). +2. **Watch ingestion + queue delay.** `watch_events_total`, `watch_restarts_total`, + `watch_replay_seconds`, `watch_replay_objects`, `watch_recovery_total`, `watch_active`, and + `watch_event_queue_seconds`. Sites: + [target_watch.go](../../internal/watch/target_watch.go), + [manager.go](../../internal/watch/manager.go). Ship Row 2. 3. **Relevance filter + git push health.** `watch_events_filtered_total`, - `git_push_duration_seconds`, `git_push_conflicts_total`. Sites: - filter decision points on the watch-to-Git path, [git_atomic_push.go](../../internal/git/git_atomic_push.go), - and [branch_worker.go](../../internal/git/branch_worker.go). Ship Row 3. -4. **Firehose + adaptive grace.** Finish opt-in `diag_all`; use per-type coverage to skip the grace - wait for never-attributed types; ship the dashboard JSON and alert rules. + `git_push_duration_seconds`, `git_push_conflicts_total`. Sites: the filter decision points on the + watch-to-Git path, [git_atomic_push.go](../../internal/git/git_atomic_push.go), and + [branch_worker.go](../../internal/git/branch_worker.go). Ship Row 3. +4. **Dashboard, alert rules, and adaptive grace.** Ship the Grafana JSON and the alert rules, and use + per-type coverage to skip the grace wait for types audit never covers — no point delaying a watch + event for a fact that never comes. The metric is the prerequisite; the optimization follows it. ## 10. Non-goals / risks - **Not** cross-pod HA aggregation — single active replica today ([architecture.md → Operational Boundaries](../architecture.md#operational-boundaries)); metrics are - per-pod and that's fine. + per-pod and that's fine. The fact stream is per-replica-followed by design, so a follower gauge is + a per-pod statement, not a cluster one. - **Not** per-mutation history — watch collapses to current state across gaps; metrics count observations, not mutations. - RV-based "watch lag" (how far behind the apiserver a watch is) is attractive but hard to compute honestly across types; deferred, not in scope. -- Keep `diag_all` opt-in/bounded — a full audit firehose in prod would bloat Redis. - Do **not** reintroduce the retired body-join metrics (`audit_join_*`, `audit_official_gate_wait`, - `parked`/`shallow_dropped` outcomes) — they belong to an architecture that no longer exists. + `parked`/`shallow_dropped` outcomes) or the v1 keyspace's `exact_deletecollection_item` — they + belong to architectures that no longer exist. +- Do **not** subtract counters across populations. `written` minus `matched` is not delivery loss: + `written` counts every fact appended for every type, while the follower files only facts on streams + **this process follows**, and a restart re-reads the retention window and files the same facts + again. Two counters over different populations do not subtract; delivery loss is measured where + delivery happens, which is what §5.4 does. ## References - [architecture.md](../architecture.md) — leading source of truth (esp. *Common Flows*, *Optional Attribution*, *State Ingestion*, *Observability*). - [interpreting-metrics.md](../interpreting-metrics.md) — the live baseline + the per-metric doc bar. -- [watch-first-ingestion-architecture.md](../finished/watch-first-ingestion-architecture.md) — the watch-first - ingestion design and the earlier metric sketch this plan modernizes. +- [attribution-metrics-proposal.md](attribution-metrics-proposal.md) — the reasoning trail for §4.4 + and §5, including what an earlier draft got wrong. +- [attribution-publish-and-join.md](attribution-publish-and-join.md) — how the two halves work, and + the tier ladder the `tier` label names. +- [attribution-fact-stream.md](../finished/attribution-fact-stream.md) — the shipped transport, the + in-process index, and the follower these metrics watch. +- [watch-first-ingestion-architecture.md](../finished/watch-first-ingestion-architecture.md) — the + watch-first ingestion design and the earlier metric sketch §4.1 modernizes. diff --git a/docs/design/open-asks-priority.md b/docs/design/open-asks-priority.md index 01e51b31..f4f35a43 100644 --- a/docs/design/open-asks-priority.md +++ b/docs/design/open-asks-priority.md @@ -13,7 +13,7 @@ > what was asked. > > One of the queue's entries is already specified rather than merely wanted: -> [`attribution-fact-stream.md`](attribution-fact-stream.md) picks and details the replacement for +> [`attribution-fact-stream.md`](../finished/attribution-fact-stream.md) picks and details the replacement for > the attribution keyspace. It is ranked here like everything else, and it changes how the > highest-priority consumer ask should be built. See > [attribution facts as a stream](#attribution-facts-as-a-stream-tier-2-and-it-answers-15s-hard-part). @@ -37,7 +37,7 @@ Four tests, applied in order. They are what produced the queue in Test 3 is why this document exists rather than a straight re-ranking of the asks, and it lands on sibling inference. It has a second instance, arrived at independently: -[`attribution-fact-stream.md`](attribution-fact-stream.md) deletes the fact keyspace and the +[`attribution-fact-stream.md`](../finished/attribution-fact-stream.md) deletes the fact keyspace and the `deletecollection` expander rather than optimizing either, and ends up with less code doing more. Two of these in one quarter is a pattern worth naming: the parts of this system that hurt are the parts that reconstruct something from state they do not own. @@ -146,7 +146,7 @@ they need a `byType` line at the moment it matters rather than by noticing a fil | 22 | `ReasonRefusedStructural` doc says "permanent"; refusal-detail stem; `Actor` reachability | gitops-api | **0** | | 15 | A declared `auditRoute` with zero facts must say so | gitops-api | **1** | | n/a | Delete sibling inference (answers #10) | this doc | **1** | -| n/a | Attribution facts become a stream; the keyspace and the expander are deleted | [`attribution-fact-stream.md`](attribution-fact-stream.md) | **2** | +| n/a | Attribution facts become a stream; the keyspace and the expander are deleted | [`attribution-fact-stream.md`](../finished/attribution-fact-stream.md) | **2** | | F6 | `spec.suspend`, `spec.interval`, `requestedAt` | maintainer review | **2** | | 5 | `CommitRequest.spec.author`, SAR-guarded | gitops-api (#220) | **2** | | B4 | `commitWindow` / `commit.message` move to GitTarget | config surface | **2** | @@ -214,14 +214,14 @@ defer #15 until the transport changes, because the stream design makes the signa to produce. The right move is the opposite: ship the condition now, and define it in terms that both transports can answer, which is "how many facts has this route contributed, and when was the last one". Today that is a counter incremented where -[`RecordFact`](../../internal/queue/attribution_index.go) writes; after the change it is the same +`RecordFact` writes; after the change it is the same counter incremented where the receiver appends. The condition never learns which transport it has, which is the same seam rule the stream design argues for one level down. Where `auditRoute` came from is [`attribution-fact-identity.md`](attribution-fact-identity.md). The related open question about *how the watch waits* is no longer open in the way it was: the six options in [`attribution-wait-poll-vs-push.md`](attribution-wait-poll-vs-push.md) are superseded by -[`attribution-fact-stream.md`](attribution-fact-stream.md), which picks one and specifies it. +[`attribution-fact-stream.md`](../finished/attribution-fact-stream.md), which picks one and specifies it. **The inference deletion** sits in this tier for the reason argued above: repo state changing operator behavior invisibly is the same class of defect as an audit route that silently resolves @@ -234,7 +234,7 @@ one coordinated bump; doing them one at a time costs six. #### Attribution facts as a stream: Tier 2, and it answers #15's hard part -[`attribution-fact-stream.md`](attribution-fact-stream.md) is the only item in this queue that +[`attribution-fact-stream.md`](../finished/attribution-fact-stream.md) is the only item in this queue that arrives already specified, and it is the largest. It is in Tier 2 rather than Tier 1 for an honest reason: today's keyspace is *slow*, not *wrong*. The poll loop runs to completion on essentially every attributable event, which is waste, and waste does not outrank a silent misconfiguration. diff --git a/docs/design/attribution-fact-stream.md b/docs/finished/attribution-fact-stream.md similarity index 70% rename from docs/design/attribution-fact-stream.md rename to docs/finished/attribution-fact-stream.md index 605af54a..4a122137 100644 --- a/docs/design/attribution-fact-stream.md +++ b/docs/finished/attribution-fact-stream.md @@ -1,9 +1,12 @@ # Attribution facts as a stream, not a keyspace -> **design**: proposed, not built. Index: [`../INDEX.md`](../INDEX.md) +> **design**: BUILT and shipped. The v1 fact keyspace and the `deletecollection` expander are gone, +> the resolver waits on the in-memory index, and the transport is selectable. See +> [what is built](#what-is-built-and-what-the-code-settled) for what each piece landed as, and for +> the numbers the implementation chose. Index: [`../INDEX.md`](../INDEX.md) > > Supersedes the option analysis in -> [`attribution-wait-poll-vs-push.md`](attribution-wait-poll-vs-push.md), which priced six ways to +> [`attribution-wait-poll-vs-push.md`](../design/attribution-wait-poll-vs-push.md), which priced six ways to > stop polling Redis. This record picks one and specifies it: the audit receiver publishes facts, > the watch side subscribes per type and holds them in memory, and the per-key Redis lookup is > deleted rather than optimized. @@ -36,9 +39,61 @@ against that index, and the grace window becomes a wait on an in-process signal against Redis. `SET`, `GET`, and the three fact key shapes in -[`attribution_index.go`](../../internal/queue/attribution_index.go) go away. Redis keeps the watch +`attribution_index.go` go away. Redis keeps the watch resume cursors and the command-author records, which are unrelated and unchanged. +## What is built, and what the code settled + +The design shipped in four pieces. The first three changed no behaviour, because the resolver still +read the v1 keys; the fourth is where the behaviour change landed, all at once, so no install is +left half on each path. + +| Piece | State | What landed | +|---|---|---| +| The design | merged, [#283](https://github.com/ConfigButler/gitops-reverser/pull/283) | this record | +| The transport seam | merged, [#284](https://github.com/ConfigButler/gitops-reverser/pull/284) | `FactPublisher` / `FactFollower`, the Redis-stream and in-memory implementations, one conformance suite over both | +| The index and the publish side | merged, [#286](https://github.com/ConfigButler/gitops-reverser/pull/286) | the four match structures, the subscription set, the waiter registry, the follower loop, and `audit_handler.go` appending one entry per type per request | +| The switch-over | merged, [#287](https://github.com/ConfigButler/gitops-reverser/pull/287) | the resolver waits on the index, the v1 keys and the collection expander are deleted, `cmd/main.go` selects the transport | + +Six things the code decided that this record only pointed at: + +**`FactStreamKey` holds a typed `schema.GroupResource`, not a rendered string.** The stream name +embeds the API-path form (`apps/deployments`) that `groupResourceKey` produces, while +`schema.GroupResource.String()` produces the reversed dotted form (`deployments.apps`). A caller +rendering its own key would publish to a stream nobody follows, with no compile error and no failing +test. Holding the type and rendering at the transport boundary makes that unrepresentable. + +**The v2 streams are deliberately not siblings of the v1 keys.** The stream key suffix is +`:author:v2:audit:` and the keyspaces are unrelated, so an install that rolls back keeps reading v1 +keys while the v2 streams age out on their own. + +**Expiry is decided on read, not only by the sweep.** A lookup checks each entry's TTL itself, so an +aged-out fact is never joined merely because the sweep has not run. `SweepInterval` bounds memory, +never correctness. + +**A trim gap is only reported when the follower was actually behind.** Detection is gated on that, +so ordinary retention ageing out an entry a caught-up follower already read is not counted as loss. + +**`ResolveAuthor` took a query struct rather than keeping its parameter list.** This record said the +signature would not change, and that turned out to be wrong: the collection tier joins on the +object's namespace and labels, and neither could be expressed in the six arguments the resolver +took. Keeping the signature would have left step 4 — the body-less collection delete, the case the +expander gave up on and the whole reason the collection tier exists — unreachable from production. +Everything the constraint was protecting held: the grace window, the blocking behaviour, the outcome +classification, and both metric names. + +**The replica count is a flag, because the process cannot see it.** The in-memory transport has to +be refused with more than one replica, and nothing in the downward API reports a Deployment's +replica count, so the chart templates `--replica-count` in from `.Values.replicaCount`. The chart's +own `validate-replica-count.yaml` still fails a chart install outright; this is the gate for +everything installed another way. + +And two the code refused to guess at until now: `cmd/main.go` stayed untouched through #284 and +#286, so the transport-selection flag and the in-memory-plus-multiple-replicas rejection landed with +the switch-over — a flag whose consumer does not exist yet is a flag nobody can test. The tuning +numbers were likewise constants and `FactIndexConfig` fields, defaulted to the values in +[the open questions](#open-questions), and became flags at the same moment. + ## What this is replacing Today the write side stores each fact under up to two of three key shapes, and the read side polls @@ -126,7 +181,7 @@ gitops-reverser:author:v2:audit:route:: The route infix stays exactly as it is today and for the same reason: an apiserver posts under one route, several `ClusterProvider`s naming one cluster share that route and therefore share its facts, and a fact from cluster A must never name the author of an object watched on cluster B. See -[`attribution-fact-identity.md`](attribution-fact-identity.md). +[`attribution-fact-identity.md`](../design/attribution-fact-identity.md). The group/resource suffix is the new part, and it is what makes the fan-out meaningful. A process watching only `configmaps` and `deployments` follows two streams and never receives a fact for @@ -139,8 +194,8 @@ anything else. decides which events produce a usable fact. The change is to accumulate rather than write immediately: -1. Each accepted event is reduced to an [`AuthorFact`](../../internal/queue/attribution_index.go#L90) - as [`RecordFact`](../../internal/queue/attribution_index.go#L132) does now, with one change: a +1. Each accepted event is reduced to an `AuthorFact` + as `RecordFact` does now, with one change: a `deletecollection` is published as **one fact describing the collection**, not expanded into one fact per object. See [collection deletes](#collection-deletes-are-one-fact). 2. Facts are grouped by `(route, group/resource)` across the whole request. @@ -156,7 +211,7 @@ anyone. The "no resolvable name" rule is the one exception that changes: a name- `deletecollection` is now exactly the case that produces a fact, so the check becomes "no name and not a collection verb". -[`AuthorFact`](../../internal/queue/attribution_index.go#L90) gains two fields: the label selector +`AuthorFact` gains two fields: the label selector from the request URI, and the optional set of uids the collection covered, reduced from the response body at the receiver and dropped past a size cap. It already carries the namespace, verb, and stage timestamp that the collection join reads, and it stops needing the per-item name and namespace the @@ -212,7 +267,7 @@ Entries are applied in stream order into four structures: | collection | `(route, group/resource, namespace)`, time-bounded, with an optional uid set | removals caused by a `deletecollection` | The first three mirror the key shapes the lookup already knows, so the join policy in -[`LookupAuthorResolution`](../../internal/queue/attribution_index.go#L382) survives unchanged. The +`LookupAuthorResolution` survives unchanged. The fourth is new and is described in [the next section](#collection-deletes-are-one-fact). **The route leads every key, and it has to.** The index is one per process while the streams are one @@ -221,7 +276,7 @@ one map and hand a watch event on cluster B an author from cluster A. The rv-onl that bites hardest, because a resourceVersion is opaque and not unique across clusters, and the collection tier is where it bites most quietly, because a namespace name says nothing about which cluster it is in. The v1 fact keys already carry the route for this reason -([`attribution-fact-identity.md`](attribution-fact-identity.md)), and the same dimension has to +([`attribution-fact-identity.md`](../design/attribution-fact-identity.md)), and the same dimension has to travel through the waiter candidate keys and the collection scope match, not only the four maps above. A test that stores identical `(group/resource, uid, rv)` facts under two routes and resolves each from its own is the one that proves it, and it belongs with the index rather than the @@ -237,19 +292,61 @@ oldest-first eviction. ### The resolver -[`ResolveAuthor`](../../internal/watch/author_resolver.go#L160) keeps its signature, its grace -window, and its blocking behavior. Only the middle changes: +[`ResolveAuthor`](../../internal/watch/author_resolver.go) keeps its grace window and its blocking +behavior. Only the middle changes: 1. Register a waiter for this event's candidate keys. 2. Check the in-memory index. 3. If absent, block on the waiter, `ctx.Done()`, or the grace deadline. -4. On a hit, return exactly as today, including the - [outcome classification](../../internal/watch/author_resolver.go#L198) that distinguishes +4. On a hit, return with the + [outcome classification](../../internal/watch/author_resolver.go) that distinguishes not-attempted from unresolved. Step 1 must precede step 2. Registering after the check loses a fact applied in the gap between them, which is the same race the poll loop currently papers over by looking again. +**A hit does not always end the wait, and assuming it did was a bug this record shipped with.** +Ordering the tiers only decides between facts that are both PRESENT. The watch event reliably beats +the audit batch carrying its delete — that is the entire reason the grace window exists — so when a +removal is resolved, the only fact in the index is often the object's last write. Returning on it +answered "who deleted this" with "who last edited it" every time someone else had touched the object +first, and no ordering could have helped, because the right fact had not been delivered yet. + +So a per-object match on a REMOVAL is held as a fallback rather than returned, unless the fact is +itself about a deletion. The wait continues for evidence about the removal — either collection tier, +or the object's own delete fact — and the fallback is returned when the grace expires with nothing +better. Waiting never costs an attribution: the worst case returns exactly what returning early +would have, one grace window later, which is the case the grace window is for. + +It does cost LATENCY, and the number belongs here rather than in a dashboard someone discovers it +from. Measured over one e2e run, by tier: a removal that finds its delete evidence resolves in about +70ms, and one that never does averages about 3.1s before falling back to the last write. Creates and +updates never consult those tiers. So the cost is not spread across attribution, it is concentrated +in removals for which no delete fact ever arrives, which is most often a type the cluster's AUDIT +POLICY excludes rather than anything Kubernetes withholds: those spend the grace to end up exactly +where they started. (An earlier draft of this said a graceful pod delete produces no audit event at +all. It does — the DELETE request is audited like any other, and under deletion-as-intent that +request is the fact the join wants. Pods are absent from this repository's own e2e facts because the +recommended policy drops them as runtime noise.) The +shard is single-threaded, so the wait also delays whatever is queued behind it, and +`--author-attribution-grace` is the one lever that bounds both. + +**What is deliberately NOT claimed here is a before-and-after.** Comparing a run of this against a +run of the previous behaviour looked easy and was not: the two runs' populations differed by more +than the change (one had 203 non-exact resolutions against the other's 59, and the specs added +alongside this work generate collection deletes that shift the mix by construction), and the +baseline was never captured per tier. A headline "the mean wait moved from X to Y" out of those two +runs would have been a workload difference wearing a causal claim's clothes. The per-tier numbers +above are from a single run and need no comparison to mean something. + +Trading the wait back is a product decision about how much commit latency a correct deletion author +is worth; the alternative on offer is naming an innocent person. + +Its parameter list does change, into an `AuthorQuery` carrying the object's namespace and labels +alongside the route, type, uid and resourceVersion it already took. Those two fields are what the +collection tier joins on, so without them a body-less `deletecollection` could never reach step 4 +from production — see [what the code settled](#what-is-built-and-what-the-code-settled). + The waiter signal comes from the goroutine applying stream entries. There is no Redis call anywhere on this path: the fast case is a map read, and the waiting case is a channel receive. @@ -266,7 +363,7 @@ independent watch events. That asymmetry is not an accident of our plumbing, it mechanisms are: audit reports the request that was made, and the watch reports each object that changed. The lab corpus records exactly this shape for row 9, watch times N against audit times one. -[`RecordDeleteCollectionFacts`](../../internal/queue/attribution_index.go#L242) tries to erase the +`RecordDeleteCollectionFacts` tries to erase the asymmetry by rebuilding the N from the one, parsing `responseObject` into a list of per-object identities and writing a fact for each. Everything about that is fragile: @@ -289,16 +386,26 @@ was given. Then a removal joins it by **scope** instead of by identity. For a removal event, the resolver tries in order: 1. the exact `(group/resource, uid, rv)` fact, -2. the `latest` fact for that uid, -3. a **collection** fact whose scope matches and whose **uid set contains this object**, when the - collection carried a usable response body, +2. a **collection** fact whose **uid set contains this object**, when the collection carried a + usable response body, +3. the `latest` fact for that uid, 4. a **collection** fact whose group/resource and namespace match the object, whose selector matches the object's labels, and whose stage timestamp is within the collection window, 5. the rv-only hatch. -Precedence is the correctness argument. A collection fact is the weakest evidence in the table, so -it only ever names an author when nothing more specific does. An unrelated delete by another actor -during the same window is claimed by its own fact at step 2 and never reaches step 4. +**The two collection tiers sit on opposite sides of `latest`, and this record originally had both +below it.** That was wrong, and the implementation corrected it. The `latest` tier answers "who last +WROTE this object"; a removal asks "who DELETED it". For a single-object delete the two coincide, +because the delete files its own fact under that uid and overwrites what was there. A collection +delete files one fact about the collection, so the uid's `latest` entry is left holding whoever +edited the object last — and ranking it above the collection's uid set credited every removal to the +previous editor while the uid set went unread. It is the one thing the expander got right, by +overwriting that entry per object. Uid membership is the API server stating that THIS request +deleted THIS object, so nothing weaker may answer ahead of it. + +Scope matching stays below `latest`, because it is the weakest evidence here and the only tier that +can name the wrong human. An unrelated delete by another actor during the same window is claimed by +its own fact at step 3 and never reaches step 4. Steps 3 and 4 are the same fact matched two ways, and the split is the subject of [the next section](#should-the-response-body-travel-with-the-fact). Step 4 alone is already more @@ -386,7 +493,7 @@ set is an opportunistic upgrade taken when the apiserver happened to send a body This applies to collection deletes only. For a normal write, the watch event already carries the full object, so an audit body would duplicate it for no gain, and the fact already extracts the one thing it needs from `responseObject`, the post-write resourceVersion -([`rvFromRawObject`](../../internal/queue/attribution_index.go#L558)). +(`rvFromRawObject`). That question is settled in this repository rather than open. Row 15 of the lab corpus established that an aggregated-API write produces an audit event with an **empty** body while the watch carries @@ -503,11 +610,22 @@ no stream reader started and no subscription taken. **The fan-in property.** One fact still serves every `GitTarget` that needs it, now through a shared index instead of a shared key. -**Facts that never resolve.** A status subresource update and a graceful pod delete produce no audit -event at all, so no wait and no transport can name their author. They still spend the grace window -and ship unresolved. That is unchanged, and it is the population that keeps -[the circuit breaker](attribution-wait-poll-vs-push.md#option-c-circuit-break-a-route-that-has-never-resolved-anything) -worth building separately. +**Facts that never resolve.** Some events produce no audit fact at all, so no wait and no transport +can name their author. They still spend the grace window and ship unresolved. That is unchanged, and +it is the population that keeps +[the circuit breaker](../design/attribution-wait-poll-vs-push.md#option-c-circuit-break-a-route-that-has-never-resolved-anything) +worth building separately — see +[when a removal should stop waiting](../design/attribution-removal-wait-options.md). + +**What that population is, corrected.** This record named "a status subresource update and a +graceful pod delete" and called them structural. They are not: this repository's e2e audit policy +drops `pods` and every `*/status` as runtime noise, and the mutation-capture lab runs against that +same cluster, so corpus rows 5 and 7 recorded the POLICY's silence rather than the API server's. A +`DELETE` request on a pod is audited like any other request, and under the deletion-as-intent rule +that request is exactly the fact the join wants. The population that can never resolve is therefore +mostly **the types the cluster's audit policy excludes** — which is configuration, is knowable, and +is a far better thing to be up against than a wall. Confirming it by measurement, rather than by +reading the policy, is worth a lab run. ## Starting up and catching up @@ -638,14 +756,20 @@ rejects that when author attribution or the admission webhook is enabled [`lookupTargetWatchCursor`](../../internal/watch/target_watch.go#L956) returns a miss and the watch cold-replays on restart, which is correct and only more expensive. -So Redis is a hard requirement for **attribution and the admission webhook**, and for nothing else. -Making attribution work without it is one seam, not a project. +So Redis is a hard requirement for **attribution**, and for nothing else. Making attribution work +without it is one seam, not a project. -That requirement narrows once the transport is selectable, and the startup validation has to narrow -with it or the in-memory mode is unreachable: an empty `--redis-addr` becomes an error only when the -**Redis** transport is selected, or when the admission webhook is enabled, and the combination of -the in-memory transport with an empty address becomes a supported configuration rather than a -rejected one. The flag, the validation in [`cmd/main.go`](../../cmd/main.go), and +**A correction to that sentence, which this record got wrong.** It claimed the admission webhook +required Redis too. It does not, and deliberately so: the webhook is `failurePolicy: Ignore` and the +controller is the real gate, so without Redis it simply no-ops command-author capture and +`CommitRequest`s claim no actor. That is a supported, degraded mode rather than a usage error, it +pre-dates this design, and a test pins it. Turning it into a startup error to match this record +would have broken a shape installs already run. The record is what changed. + +The remaining requirement narrows once the transport is selectable, and the startup validation has +to narrow with it or the in-memory mode is unreachable: an empty `--redis-addr` becomes an error +only when the **Redis** transport is selected, and the combination of the in-memory transport with +an empty address becomes a supported configuration rather than a rejected one. The flag, the validation in [`cmd/main.go`](../../cmd/main.go), and [`configuration.md`](../configuration.md) move together in that change, because a flag whose validation and documentation disagree is how a mode ends up unreachable in the first place. @@ -704,19 +828,20 @@ restart today, which is what keeps the delta small. ## Code inventory -| Change | Where | -|---|---| -| Delete the fact key builders, `SET`/`GET` paths, and the SCAN-based size gauge | [`attribution_index.go`](../../internal/queue/attribution_index.go) | -| Delete the collection expander: `RecordDeleteCollectionFacts`, `storeDeleteCollectionFacts`, `deleteCollectionItems`, `deleteCollectionItem`, and their tests | [`attribution_index.go`](../../internal/queue/attribution_index.go), `attribution_index_deletecollection_test.go` | -| Retire §5 and §8 of the expander spec; §2, the deletion-as-intent render rule, is untouched and still binds | [`deletecollection-attribution-expander.md`](../spec/deletecollection-attribution-expander.md) | -| Group per request, one `XADD` per type | [`audit_handler.go`](../../internal/webhook/audit_handler.go) | -| New: the two-method transport seam, plus a Redis-stream and an in-memory implementation and one conformance suite over both | new files under [`internal/queue/`](../../internal/queue/) | -| New: subscription set, in-memory index, waiter registry, all transport-agnostic | new file under [`internal/queue/`](../../internal/queue/) | -| Add the transport selection flag and reject in-memory with more than one replica | [`cmd/main.go`](../../cmd/main.go) | -| Correct the stale "hard dependency in every mode" comment | [`redis_store.go`](../../internal/queue/redis_store.go#L45) | -| Wait on a signal instead of polling; drop `attributionPollInterval` | [`author_resolver.go`](../../internal/watch/author_resolver.go) | -| Subscribe and unsubscribe a type as watches come and go | [`target_watch.go`](../../internal/watch/target_watch.go) | -| Replace the fact-index size gauge; add eviction and trim-gap counters | [`telemetry/exporter.go`](../../internal/telemetry/exporter.go) | +| Change | Where | State | +|---|---|---| +| New: the two-method transport seam, plus a Redis-stream and an in-memory implementation and one conformance suite over both | [`fact_stream.go`](../../internal/queue/fact_stream.go), [`fact_stream_redis.go`](../../internal/queue/fact_stream_redis.go), [`fact_stream_memory.go`](../../internal/queue/fact_stream_memory.go) | done, #284 | +| New: subscription set, in-memory index, waiter registry, all transport-agnostic | [`fact_index.go`](../../internal/queue/fact_index.go), [`fact_index_store.go`](../../internal/queue/fact_index_store.go), [`fact_streams.go`](../../internal/queue/fact_streams.go), [`fact_waiters.go`](../../internal/queue/fact_waiters.go) | done, #286 | +| Group per request, one append per type; publish a collection as one fact | [`audit_handler.go`](../../internal/webhook/audit_handler.go), [`author_fact.go`](../../internal/queue/author_fact.go) | done, #286 | +| Add the eviction, trim-gap, and collection-degraded counters | [`telemetry/exporter.go`](../../internal/telemetry/exporter.go) | done | +| Wait on a signal instead of polling; drop `attributionPollInterval` | [`author_resolver.go`](../../internal/watch/author_resolver.go) | done | +| Subscribe and unsubscribe a type as watches come and go | [`target_watch.go`](../../internal/watch/target_watch.go) | done | +| Delete the fact key builders, `SET`/`GET` paths, and the SCAN-based size gauge. The file goes with them: what survived is the fact shape and the result taxonomy, in [`author_fact.go`](../../internal/queue/author_fact.go), and the shared key helpers, in [`key_prefix.go`](../../internal/queue/key_prefix.go). `attribution_fact_index_size` survives too, now a field read on the sweep rather than a SCAN of the whole keyspace | `attribution_index.go`, deleted | done | +| Delete the collection expander: `RecordDeleteCollectionFacts`, `storeDeleteCollectionFacts`, and their tests. `deleteCollectionItems` survives, reduced to uids only: the publish side still needs the body parsed once, to build the uid SET one fact carries. Nothing rebuilds N per-object facts from one request | `attribution_index.go` and `attribution_index_deletecollection_test.go`, deleted | done | +| Retire §5 and §8 of the expander spec; §2, the deletion-as-intent render rule, is untouched and still binds | [`deletecollection-attribution-expander.md`](../spec/deletecollection-attribution-expander.md) | done | +| Add the transport selection flag, narrow the `--redis-addr` validation, and reject in-memory with more than one replica | [`cmd/main.go`](../../cmd/main.go), [`configuration.md`](../configuration.md) | done | +| Correct the stale "hard dependency in every mode" comment | [`redis_store.go`](../../internal/queue/redis_store.go) | done | +| Document the three new counters and the replaced result label | [`interpreting-metrics.md`](../interpreting-metrics.md) | done | `AttributionResolutionsTotal` and `AttributionResolutionWaitSeconds` keep their names and meanings, so the e2e reporting in [`reportAttributionStats`](../../test/e2e/e2e_suite_test.go) and every @@ -736,14 +861,24 @@ already true of any restart today. ### What a fact holds about a person A fact names an actor, so it carries personal data and should be described as such: the username, -and the display name and email when the API server supplied them, alongside the object identity, -verb, and stage timestamp. That is the same content the v1 fact keys hold today, taken from the -audit event's `user` field, and moving it from a key to a stream entry changes where it lives rather -than what it is. +and the display name and email when the API server supplied them, alongside the object's namespace +and uid, the verb, and the stage timestamp. That is taken from the audit event's `user` field, and +moving it from a key to a stream entry changed where it lives rather than what it is. + +It carries **less** than the v1 keys did. The switch-over dropped the object's name and subresource +and the group/resource from the wire, because no join tier reads them — the type is the stream's own +name, and the join is by uid, resourceVersion, or scope. `isServiceAccount` went too, being a prefix +check on the username rather than evidence. A fact is broadcast to every process following its type, +held for the whole TTL, and replayed into memory on every restart, so a field nothing reads is paid +for three times; on a real collection-delete fact the removals cut the entry by about a quarter. The +one field kept without being read is `auditID`, which is what ties a commit authored by the wrong +person back to the audit event that named them. Retention moves the same way. A fact is held for `--author-attribution-ttl` (ten minutes by default) in the Redis stream and, once read, in the process's in-memory index, and the trim and the TTL sweep -drop it after that. Nothing writes it to Git: the commit carries the author's +drop it after that. The stream KEY carries the same deadline, refreshed by every append, so a type +that stops being written to takes its stream with it instead of leaving an immortal key behind; the +in-memory transport forgets an idle ring on the same horizon. Nothing writes it to Git: the commit carries the author's name and email as commit metadata, which is what the actor already published by making the change. Access is whoever can read the Redis keyspace and the pod's memory, which is why the keyspace is namespaced per install ([`--redis-key-prefix`](../configuration.md)) and why an install that @@ -767,24 +902,37 @@ would build a compatibility path for a topology the chart refuses to start. ## Open questions -- **How large may a collection's uid set be before it is dropped?** It bounds one entry's size, the - broadcast to every subscriber, and the replay on restart. The fallback is already correct, so this - is a tuning number rather than a correctness one, but it decides how often the precise path is - taken. -- **How long is the collection window?** Short enough that an unrelated delete in the same namespace - is not claimed, long enough to cover audit batching plus clock skew. The deletion-as-intent rule - means it does not have to cover finalization, so a few multiples of the grace window is the - starting point. It should be measurable from the lab's per-scenario timing report. -- **What replaces the `exact_deletecollection_item` result label?** The match is now scope-based, so - the name is wrong. Renaming it is a visible change to an exported metric's label value, which - needs saying in the release notes even though the metric keeps its name. -- **Cap per type, or one global cap?** Per type is fairer under a burst on one noisy type. A global - cap is simpler and bounds the pod. Probably both, with the per-type cap as the primary. +### Answered by the implementation + +Each of these is a constant in [`internal/queue/`](../../internal/queue/) today and becomes a flag +in the switch-over, when a consumer for it first exists. + +| Question | Answer | Why that number | +|---|---|---| +| How large may a collection's uid set be? | **10000 uids** (`DefaultCollectionUIDCap`) | A few hundred kilobytes against a body for the same request that runs to tens of megabytes. A collection larger than that is exactly the one a cluster with audit truncation enabled would not have sent a body for anyway. | +| How long is the collection window? | **30s** (`DefaultFactCollectionWindow`) | Ten times the default grace window. Under the deletion-as-intent rule it only has to cover audit batching plus clock skew, so it can be far shorter than the fact TTL, and short enough that an unrelated delete a minute later is not claimed. | +| Cap per type, or one global cap? | **Both**: 4096 per `(route, group/resource)`, 65536 total | Per-type is the primary because it is the fair one — a burst on one noisy type must not evict every other type's facts. The total cap bounds the pod with a number that does not scale with how many types happen to be watched; overflow evicts from the type holding the most, so the pressure lands where it came from. | +| `BLOCK` interval | **1s** (`DefaultFactStreamBlock`) | A follower re-reads the subscription set on every `Next`, so this is also how long a subscribe or unsubscribe takes to land. | + +**What replaces the `exact_deletecollection_item` result label**: two labels rather than one, because +the match is now two-tiered. `collection_uid` is a removal whose uid was in the set the API server +said it deleted — no over-attribution risk at all — and `collection_scope` is one matched by +namespace, selector, and window alone, which is the weaker evidence. `exact_deletecollection_item` +disappears with the expander in the switch-over, and that is the visible metric-label change the +release notes have to carry. + +Three counters were added alongside them, registered with the index and first emitted when it is +wired in: `attribution_fact_index_evictions_total{reason}` (`per_type` or `total`), +`attribution_fact_stream_gaps_total{stream}`, and `attribution_collection_degraded_total{reason}` +for a collection fact published without its uid set. They get their row in +[`interpreting-metrics.md`](../interpreting-metrics.md) in the switch-over, where they start moving. + +### Still open + - **Is one entry per type per request the right granularity?** It matches the apiserver's batching exactly, but a single entry can then carry hundreds of facts. An entry-size ceiling that splits - oversized groups may be needed. -- **`BLOCK` interval.** It sets how quickly a subscription change takes effect and how often the - reader wakes for nothing. Likely a second, worth confirming against the observed publish rate. + oversized groups may be needed. `DefaultFactStreamMaxLen` bounds a stream in *entries*, not bytes, + so nothing bounds one entry's size today. - **Does the per-type stream count stay reasonable?** One `XREAD` across a few dozen streams is ordinary. A cluster watching several hundred types would want checking before it is assumed. - **Should the trim-gap counter feed a condition?** A reader that is losing facts is a real diff --git a/docs/finished/redis-key-schema-v3.md b/docs/finished/redis-key-schema-v3.md index 41dd5f32..5a363d7d 100644 --- a/docs/finished/redis-key-schema-v3.md +++ b/docs/finished/redis-key-schema-v3.md @@ -21,7 +21,7 @@ > fallback the `rv:` key would be dead. `rv:` is type-scoped and per-write, so consulting it does not > reintroduce the stale-LWW hazard `:last` would. > Related: -> [`internal/queue/attribution_index.go`](../../internal/queue/attribution_index.go), +> `internal/queue/attribution_index.go`, > [`internal/queue/redis_store.go`](../../internal/queue/redis_store.go), > [`internal/watch/author_resolver.go`](../../internal/watch/author_resolver.go), > [CommitRequest authorship from admission](../spec/commitrequest-admission-authorship.md), @@ -45,7 +45,7 @@ the retired `poc/redis-copy` line. Those are not live Redis owners in this branc not assign them new keys. The audit author family writes **up to three keys per audit event**, strongest first -([`factKeyVariants`](../../internal/queue/attribution_index.go#L429)): +(`factKeyVariants`): - `e` — **exact**: `(group, resource, ns, name, uid, rv)` — only when both uid *and* rv are known. - `u` — **uid-only**: `(group, resource, ns, name, uid)` — only when uid is known. @@ -149,7 +149,7 @@ func groupResourceKey(group, resource string) string { ``` Group/resource and a UUID never contain `:` or `/`, so the per-field escaping -([`joinKeyFields`](../../internal/queue/attribution_index.go#L524)) that v2 needed for RBAC names like +(`joinKeyFields`) that v2 needed for RBAC names like `system:node-proxier` in the *name* field is no longer load-bearing for attribution (the name is a value now). RVs are numeric in practice; treat them as opaque and reject/escape a stray delimiter defensively. @@ -171,7 +171,7 @@ A given `(uid, rv)` had **exactly one writer** — that RV exists *because* of t `object::` is **written once and never contended**. That dissolves the machinery v2 needed: - **v2:** the shared `u` (uid-only) key is written by every author of the object, so a second, different - author collapses it to a `{conflict:true}` marker ([`storeFactKey`](../../internal/queue/attribution_index.go#L457)), + author collapses it to a `{conflict:true}` marker (`storeFactKey`), and only the `e` (exact) key rescues precise per-write credit. - **v3:** each write owns its own immutable `:` key, so there is **nothing to conflict**. The resolver rule becomes **"exact for exact-capable events; `:last` only for known RV-mismatch events."** @@ -245,7 +245,7 @@ disappears because `:seen` tombstones are gone (§4.2). Net dashboard change: dr ## 7. deletecollection stops being a special case at the key layer -[`RecordDeleteCollectionFacts`](../../internal/queue/attribution_index.go#L205) exists to write "only the +`RecordDeleteCollectionFacts` exists to write "only the uid-only variant" because the body RV is dead. In v3 that is simply "write `object::last`" — the same key any RV-mismatch event uses. The only deletecollection-specific thing left is the **reason code**, now driven by `fact.verb` in the value, not by which key variant matched. diff --git a/docs/interpreting-metrics.md b/docs/interpreting-metrics.md index a0eb34b8..019884a4 100644 --- a/docs/interpreting-metrics.md +++ b/docs/interpreting-metrics.md @@ -1,6 +1,6 @@ # Interpreting GitOps Reverser Metrics -> Last updated: June 2026 — reconciled to the live instrument set in +> Last updated: July 2026 — reconciled to the live instrument set in > [`internal/telemetry/exporter.go`](../internal/telemetry/exporter.go). This is the operator's field guide to the metrics GitOps Reverser exports. It explains how to @@ -155,11 +155,15 @@ sum by (group, version, resource) (rate(gitopsreverser_resync_sweep_deletes_tota ## Audit attribution (optional) -Audit runs **only when Redis is configured**. The kube-apiserver POSTs audit `EventList` -payloads to `/audit-webhook`; the handler applies an intrinsic accept gate and, for an accepted -event, writes a minimal attribution fact to the Redis index. There is **no body join and no -second source** — watch, not audit, carries the object body — so the only audit metrics are the -request boundary and the per-event census. Background: +Audit runs when `--author-attribution` is on. The kube-apiserver POSTs audit `EventList` payloads to +`/audit-webhook`; the handler applies an intrinsic accept gate and appends the accepted events' +facts to a per-type fact log — **one append per type per request**, not one per event. The watch side +follows that log into a bounded, TTL'd in-memory index and joins against it. There is **no body join +and no second source** — watch, not audit, carries the object body — so the only audit metrics are +the request boundary and the per-event census. + +The log is Redis Streams by default and an in-process ring with +`--author-attribution-transport=memory`, which is why attribution no longer implies Redis. Background: [architecture.md → Optional Attribution](architecture.md#optional-attribution). | Metric | Type | Labels | @@ -168,10 +172,17 @@ request boundary and the per-event census. Background: | `audit_eventlist_events_total` | counter | `outcome` | | `audit_eventlist_duration_seconds` | histogram | `outcome` | | `audit_events_total` | counter | `outcome`, `category`, `group`, `version`, `resource`, `verb` | -| `attribution_resolutions_total` | counter | `result`, `group`, `version`, `resource` | -| `attribution_resolution_wait_seconds` | histogram | `result` | -| `attribution_fact_events_total` | counter | `op` | -| `attribution_fact_index_size` | gauge | — | +| `attribution_resolutions_total` | counter | `tier`, `actor_kind`, `group`, `version`, `resource` | +| `attribution_resolution_wait_seconds` | histogram | `tier`, `event_kind`, `group`, `version`, `resource` | +| `attribution_facts_total` | counter | `op` | +| `attribution_fact_index_entries` | gauge | — | +| `attribution_fact_index_evictions_total` | counter | `reason` | +| `attribution_fact_stream_gaps_total` | counter | `stream` | +| `attribution_fact_stream_decode_errors_total` | counter | `transport` | +| `attribution_fact_follower_errors_total` | counter | `transport` | +| `attribution_fact_follower_last_success_timestamp_seconds` | gauge | — | +| `attribution_collection_without_uidset_total` | counter | `reason` | +| `attribution_transport_info` | gauge (always 1) | `transport` | **EventList request boundary.** `audit_eventlists_total` and `audit_eventlist_duration_seconds` count requests at `/audit-webhook`; `audit_eventlist_events_total` counts the decoded event items @@ -184,13 +195,25 @@ selector): | `category` | Live `outcome` values | Meaning | | --- | --- | --- | -| `stored` | `queued` | Accepted; an attribution fact was written to the index. | -| `dropped` | `nil_event`, `stage`, `read_only_or_unknown_verb`, `failed_request`, `dry_run`, `unchanged_resource_version`, `non_scale_subresource` | Correctly rejected at the accept gate — not an error. | -| `error` | `write_error` | The fact store rejected the write. The one category that should stay zero. | +| `stored` | `queued` | Accepted; the event's facts reached the fact log. | +| `dropped` | `nil_event`, `stage`, `read_only_or_unknown_verb`, `failed_request`, `dry_run`, `unchanged_resource_version`, `non_scale_subresource`, `no_attribution_fact` | Correctly rejected at the accept gate, or accepted and unable to name an author — not an error. | +| `error` | `write_error` | The transport rejected the append for THIS event's stream, so its fact did not reach the log. Publication is per stream, so a request that fails partway still counts the events whose own stream appended as `queued`; the whole request is failed so the API server retries it, and the landed facts are simply appended again. The one category that should stay zero. | The full enum lives in [`internal/audit/outcome/outcome.go`](../internal/audit/outcome/outcome.go) — it is the source of truth. +**`no_attribution_fact` is the one to read carefully.** The event was accepted and could name no +author, so nothing was appended for it and no watch event can ever join it. The usual cause is an +aggregated API: the kube-apiserver proxies the request and never decodes the response, so a CREATE's +`objectRef` carries no name at all. It is `dropped` rather than `error` because nothing failed, and +this is the only place it can be counted — the event never becomes a fact, so no fact-side counter +sees it. A rising share for a type you expect to attribute means commits for that type will be +authored unresolved: + +```promql +sum by (group, resource) (rate(gitopsreverser_audit_events_total{outcome="no_attribution_fact"}[5m])) +``` + **Is audit attribution alive?** Any positive rate means events are flowing: ```promql @@ -253,41 +276,167 @@ histogram_quantile(0.95, actor (human or service account) rather than producing an explicit unresolved author: ```promql -sum(rate(gitopsreverser_attribution_resolutions_total{result=~"exact_.*|weak"}[5m])) +sum(rate(gitopsreverser_attribution_resolutions_total{tier!="absent"}[5m])) / sum(rate(gitopsreverser_attribution_resolutions_total[5m])) ``` -`result` is bounded: +Coverage is `tier!="absent"` and nothing narrower. `tier=~"exact.*"` reads the collection and name +tiers as misses, which they are not — they named an actor. + +The two labels answer two different questions. **`tier`** names which evidence produced the author, +and it is ordered, strongest first. **`actor_kind`** names who that evidence named, in the same +vocabulary `commits_total{author_kind}` uses. -| `result` | Meaning | +| `tier` | Meaning | | --- | --- | -| `exact_user` | Exact UID+resourceVersion match for a human user. | -| `exact_serviceaccount` | Exact UID+resourceVersion match for a named service account. | -| `weak` | Non-exact match, such as UID-only or RV-only. | +| `exact` | Exact UID+resourceVersion match: this actor produced this exact version. | +| `collection_uid` | A removal whose UID was in the set the API server said a `deletecollection` deleted. No over-attribution risk: either the object was in that set or it was not. It outranks `latest`, because `latest` names whoever last *wrote* an object while a removal asks who *deleted* it. | +| `latest` | The UID-latest tier: the object's own last fact, keyed by UID alone. A removal consults it, and a match here describing a *write* is held as a fallback while the wait continues for evidence about the deletion. | +| `name` | A match on `(namespace, name)` for a fact carrying neither a UID nor a resourceVersion — the usual shape of an aggregated API's audit event, and of a delete the API server answered with a `Status`. | +| `collection_scope` | A removal matched to a `deletecollection` by scope alone — same type and namespace, the request's selector accepting the object's labels, within the collection window. The weakest evidence the join has, and the only one that can name the wrong actor, which is why it is reached only when every more specific tier missed. | +| `resource_version` | The RV-only escape hatch: a fact that carried a resourceVersion and no UID, matched on that version alone. | | `absent` | No usable fact matched before the grace window elapsed. The resulting live commit is authored as `unknown (attribution unresolved)`. | -**Is the grace window paying for itself?** Misses waiting near the configured grace window mean the -operator is delaying commits without finding facts: +| `actor_kind` | Meaning | +| --- | --- | +| `user` | A human (or any non-service-account subject). | +| `serviceaccount` | A named service account — a controller, an operator, a CI identity. | +| `none` | Nobody was named, which in practice means nothing matched. | + +`actor_kind="none"` and `tier="absent"` go together, and that is an invariant rather than a +coincidence: an audit event whose user cannot be resolved never becomes a fact at all (it is counted +`no_attribution_fact` above), and a fact that names nobody is refused when the follower reads it +(counted as a stream decode error below). Every fact that reaches the index therefore names someone, +which is why coverage can be read off the tier alone. + +**Evidence quality, independently of coverage.** A shift from `exact` toward `collection_scope` or +`name` is a quality regression even while coverage holds flat, so it is worth its own panel: + +```promql +sum by (tier) (rate(gitopsreverser_attribution_resolutions_total[5m])) +``` + +> **`result` is gone**, and so are `exact_user`, `exact_serviceaccount`, and `weak`. See +> [`UPGRADING.md`](UPGRADING.md) for the old-to-new mapping. `exact_deletecollection_item` went +> earlier, with the expander and the fact keyspace; `collection_uid` is its closest equivalent and +> `collection_scope` is new capability rather than a rename. See +> [`attribution-fact-stream.md`](finished/attribution-fact-stream.md). + +**Is the grace window paying for itself?** `event_kind` is `write` or `removal`, and the split is +the point: a removal holds a fallback and keeps waiting for evidence about the deletion, a write +does not. The removal wait is the number `--author-attribution-grace` is tuned from, and a p95 +approaching the configured grace means removals are sitting out the whole window: ```promql histogram_quantile(0.95, - sum by (le) ( - rate(gitopsreverser_attribution_resolution_wait_seconds_bucket{result="absent"}[5m]))) + sum by (le, tier) ( + rate(gitopsreverser_attribution_resolution_wait_seconds_bucket{event_kind="removal"}[5m]))) +``` + +**Is the fact index healthy?** Facts should be written and later matched; a high rate of +`op="written"` alongside `tier="absent"` points at a timing, type, or audit-route mismatch between +audit and watch. The two ops are **not subtractable**: `written` counts every type, `matched` only +the streams this process follows, and a restart re-files the whole retention window. + +```promql +sum by (op) (rate(gitopsreverser_attribution_facts_total[5m])) +``` + +```promql +gitopsreverser_attribution_fact_index_entries ``` -**Is the Redis fact index healthy?** Facts should be written and later matched; a high rate of -`op="written"` alongside `result="absent"` points at a timing, key, or source-identity mismatch -between audit and watch: +**Is the index dropping facts under load?** The index is bounded per type and in total, and an +attribution lost to a full index has to look different from one that was never published. Any +sustained rate here means a burst is outrunning the caps, and +`--author-attribution-max-facts-per-type` / `--author-attribution-max-facts` are the levers: ```promql -sum by (op) (rate(gitopsreverser_attribution_fact_events_total[5m])) +sum by (reason) (rate(gitopsreverser_attribution_fact_index_evictions_total[5m])) ``` +`reason` is `per_type` (one type is hotter than its share) or `total` (the whole index is under +pressure, and eviction falls on the type holding the most). + +**Is a follower losing facts?** A trim gap means the fact log was trimmed past this process's +position: the facts in the gap are gone for good, and the commits that needed them are authored +unresolved. It is the one loss the transport can see, which is why the transport is a log with +positions rather than fire-and-forget publish and subscribe. This should be **zero**: + ```promql -gitopsreverser_attribution_fact_index_size +sum by (stream) (rate(gitopsreverser_attribution_fact_stream_gaps_total[5m])) ``` +**Is a follower silently skipping facts?** An entry the follower refuses is skipped and its +position passed — which is right, since it can never decode and stalling on it would cost every +later fact on that stream — so the facts it carried are gone. This is the loss path with **no other +symptom**: unlike a trim gap it is not detectable after the fact, and unlike a publish failure the +API server does not retry it. + +Two things are refused: a payload that is not valid JSON, and one that is JSON but breaks the fact +contract by naming nobody (`author` must be present and non-empty — a fact exists to name somebody). +Either way the log line names the stream and the entry. Any non-zero rate means something is writing +entries this operator would not write — a version skew, another producer, or a hand-edited stream: + +```promql +sum by (transport) (rate(gitopsreverser_attribution_fact_stream_decode_errors_total[5m])) +``` + +**Is the fact follower alive?** The follower retries a transport failure with a backoff rather than +returning, so a wedged one degrades attribution to committer-authored across the board with a rising +unresolved rate as the only symptom. The timestamp matters more than the counter: it is the only +thing that separates "erroring occasionally while making progress" from "has read nothing in ten +minutes", and only the second is an outage. Read it as seconds since the last successful read +(idle rounds count as reads, so a quiet cluster reads as healthy): + +```promql +time() - gitopsreverser_attribution_fact_follower_last_success_timestamp_seconds +``` + +```promql +sum by (transport) (rate(gitopsreverser_attribution_fact_follower_errors_total[5m])) +``` + +**An absent gauge is the worst case, not a healthy one.** The timestamp is not published until the +follower's first successful read, so a follower that has been wedged since startup — a transport +unreachable at boot — has no series at all, and `time() - ` therefore returns nothing rather +than a large number. An alert must cover that arm explicitly, which is what `attribution_transport_info` +is for: it is published when the follower starts, so a transport running without a last-success +timestamp is exactly the never-succeeded case. Give it a `for: 10m` so an ordinary restart's gap does +not trip it: + +```promql +(time() - gitopsreverser_attribution_fact_follower_last_success_timestamp_seconds > 600) +or +(gitopsreverser_attribution_transport_info == 1 + unless on() gitopsreverser_attribution_fact_follower_last_success_timestamp_seconds) +``` + +**Which transport is in force?** `gitopsreverser_attribution_transport_info` is an info gauge whose +value is always 1, labelled `redis` or `memory`. It is a legend rather than a threshold, and it +changes how every metric above reads: a burst of unresolved commits after a restart is *expected* +under the in-process transport, which drops every fact with the process, and a *bug* under Redis. + +```promql +gitopsreverser_attribution_transport_info +``` + +**How often does a collection delete fall back to scope matching?** A `deletecollection` fact +carries the UIDs the API server named, when it sent them, and joins by membership. When it cannot, +the join falls back to `collection_scope`, which is correct but weaker — so the fallback is counted +rather than inferred: + +```promql +sum by (reason) (rate(gitopsreverser_attribution_collection_without_uidset_total[5m])) +``` + +`reason` is `uid_cap` (the set was larger than `--author-attribution-collection-uid-cap`) or +`no_uids` (the API server sent a body with no usable UIDs). A body-less response — a truncated, +aggregated, or metadata-only `deletecollection` — produces no fact-level degradation event at all, +because there was never a set to drop; it simply resolves through the scope tier. A production +cluster running `--audit-webhook-truncate-enabled` is the one most likely to be in that case. + --- ## API resource catalog @@ -374,6 +523,8 @@ rate(gitopsreverser_secret_encryption_attempts_total[5m]) | --- | --- | | `rate(gitopsreverser_audit_events_total{category="error"}[10m]) > 0` | Attribution fact-store writes are failing — check Redis. | | `rate(gitopsreverser_audit_eventlists_total{outcome="decode_error"}[10m]) > 0` | A sender is posting non-EventList payloads to `/audit-webhook`. | +| `rate(gitopsreverser_attribution_fact_stream_decode_errors_total[10m]) > 0` | A schema or version mismatch on the fact stream; facts are being skipped and lost. | +| `(time() - …_fact_follower_last_success_timestamp_seconds > 600) or (…_transport_info == 1 unless on() …_fact_follower_last_success_timestamp_seconds)`, `for: 10m` | The fact follower is wedged; attribution is degrading to committer-authored cluster-wide. Both arms are needed — see below. | | `rate(gitopsreverser_resync_background_failures_total[15m]) > 0` sustained | Background resyncs are not committing; the folder relies on steady-state events to catch up. | | `gitopsreverser_api_catalog_group_versions{state="degraded"} > 0` | Part of the API surface is hidden behind a broken APIService. | | `rate(gitopsreverser_secret_encryption_failures_total[10m]) > 0` | Secret writes are being rejected by the encryption path. | @@ -391,10 +542,23 @@ them — with a reference dashboard and an audit/attribution deep-dive — is - **Watch ingestion** — per-type watch events received, reconnects/restarts, `sendInitialEvents` replays, `410 Gone` rebuilds, and cursor-resume vs full-replay. Watch is the object-state source, yet it has almost no direct coverage today; this is the biggest gap. +- **Shard processing delay** — how long an event waits between arriving on its watch shard and being + picked up. The wait histogram above times each resolution in isolation; it cannot see the delay a + slow resolution imposes on the events queued behind it on the same single-threaded shard, which is + a failure mode that has already broken a test. +- **Relevance filter** — how many watch events are filtered before Git, and why. The filter + decisions are scattered along the watch-to-Git path today, so there is no one honest boundary to + count at yet. - **Git push health** — push latency and conflict-retry counts. The instruments for these were removed because nothing recorded them; re-add them **with** a recording site when the need is real, not before. +The attribution relabel and the fact-pipeline loss paths that used to be listed here have **shipped** +— `tier` plus `actor_kind`, `event_kind` on the wait histogram, the stream decode-error counter, and +follower health are all documented above. +Nothing consumes these metrics yet, so the break is taken deliberately in one release rather than +twice; it will carry an [`UPGRADING.md`](UPGRADING.md) table of old name → new name. + --- ## Adding a new metric to this document diff --git a/docs/spec/deletecollection-attribution-expander.md b/docs/spec/deletecollection-attribution-expander.md index f3eccba4..ce52d925 100644 --- a/docs/spec/deletecollection-attribution-expander.md +++ b/docs/spec/deletecollection-attribution-expander.md @@ -2,22 +2,25 @@ > **spec** — current behaviour. The code depends on this document; change one, change the other. Index: [`../INDEX.md`](../INDEX.md) > -> Status: **IMPLEMENTED** — 2026-06-28 (rev. 2: deletion-as-intent reframe). The render rule (§2), the -> expander (§5), the `exact_deletecollection_item` reason code (§8), unit tests (§9.1), and the four e2e -> specs (§9.2) have all landed and pass `task lint`/`task test`/`task test-e2e`. +> Status: **PARTLY SUPERSEDED.** The render rule (§2) is current behaviour and still binds — it is +> what makes the collection window short enough to be safe. The **expander (§5) and its +> `exact_deletecollection_item` reason code (§8) are DELETED**, replaced by one collection fact that +> every removal in its scope joins; §5, §6 and §8 have been rewritten to say what took their place, +> and [`attribution-fact-stream.md`](../finished/attribution-fact-stream.md) is the record that +> argues it. > Scope: two complementary pieces — (1) a render-layer rule that treats `deletionTimestamp` as **logical -> absence** and removes the file at delete-request time, and (2) the **DeleteCollection attribution expander** -> that lets a name-less collection delete be attributed per object. State correctness for collection deletes is +> absence** and removes the file at delete-request time, and (2) attribution for a name-less collection +> delete. State correctness for collection deletes is > already solved by construction in watch-first (one watch event per object); this doc adds the *intent > semantics* and the *attribution*. > Related: +> [attribution facts as a stream](../finished/attribution-fact-stream.md), > [watch-first ingestion architecture](../finished/watch-first-ingestion-architecture.md), > watch-first merge readiness §4, -> [superseded `deletecollection` nudge plan](deletecollection-attribution-expander.md), > [watch event ordering & attribution grace](../facts/watch-event-ordering-and-attribution-grace.md), > [`internal/watch/target_watch.go`](../../internal/watch/target_watch.go), > [`internal/sanitize/sanitize.go`](../../internal/sanitize/sanitize.go), -> [`internal/queue/attribution_index.go`](../../internal/queue/attribution_index.go), +> [`internal/queue/fact_index.go`](../../internal/queue/fact_index.go), > [`internal/webhook/audit_handler.go`](../../internal/webhook/audit_handler.go). ## 1. The question, and the reframe @@ -142,62 +145,37 @@ variants are skipped when no RV is supplied, which is precisely right since the - **State is solved by construction** — N watch events, mark-and-sweep backstop (merge-readiness §4). -- **A name-less event stores nothing today.** `RecordFact` early-returns when `identity.Name == ""` - ([attribution_index.go:216](../../internal/queue/attribution_index.go#L216)), so a `deletecollection` writes - **zero** facts now — the expander is purely **additive**. -- **Single deletes already attribute** (including finalizer ones, now improved). A single `kubectl delete foo` - has a name, so `RecordFact` already writes its uid-only fact; with §2's intent rule, the finalizer single-delete - is now removed and attributed at intent time too — for free, no expander needed. The expander exists **only** - for the name-less collection case. -- **The conservative resolver fails closed** — multiple authors on one key → no usable attribution fact → the - explicit unresolved author ([storeFactKey](../../internal/queue/attribution_index.go#L403)). Governing rule: +- **A name-less event produces ONE fact about the collection.** *(This bullet described the opposite + when the expander existed: a name-less event stored nothing, and the expander was purely additive. + The name check is now "no name AND not a collection verb", so the collection request is exactly the + case that produces a fact.)* +- **Single deletes attribute through their own fact** (including finalizer ones). A single + `kubectl delete foo` has a name, so it files a per-object fact under its uid; with §2's intent rule + the finalizer single-delete is removed and attributed at intent time too. Collection facts exist + **only** for the name-less collection case, and a removal reaches them only when no per-object fact + about the deletion applies. +- **The conservative resolver fails closed** — no usable attribution fact → the explicit unresolved + author. Governing rule: **a wrong author is worse than no author.** - **The grace window** absorbs a watch event that arrives before its audit fact ([author_resolver.go:40](../../internal/watch/author_resolver.go#L40)). -## 5. The expander — body-present per-UID fan-out +## 5. RETIRED — the expander is deleted -**Trigger.** An accepted, mutating audit event with `verb == deletecollection` whose response body parses as a -list of objects (typed `…List`, generic `v1.List`, or items array) — the common case for etcd-backed core types -and CRDs captured at `level: RequestResponse`. +**This section described the per-UID fan-out expander, which no longer exists.** It was deleted with +the switch to the attribution fact stream; see +[`attribution-fact-stream.md`](../finished/attribution-fact-stream.md), which argues at length why +rebuilding N per-object facts from one request was the wrong shape. -**Action.** For **every** item in the body — including finalizer-pending ones — write **one uid-only fact** keyed -on the item's own `(group, resource, namespace, name, uid)`, carrying the audit event's actor -(`resolveUserInfo`), `auditID`, `stageTimestamp`, and `Verb: "deletecollection"`. Use the *item's* namespace/name, -never the collection URL's coarse/empty ones. No skipping: a finalizer item is removed-as-intent (§2) and -attributed to the actor exactly like any other item. +What replaces it, in one line: a `deletecollection` is published as **one fact describing the +collection** — actor, type, namespace, the selector the request URI expressed, the stage timestamp, +and the set of uids the API server named when it sent a body — and every removal in that scope joins +it. The join tries uid membership first, then scope. -**Why it's honest.** Each fact names a specific UID the API server confirmed this actor issued a collection delete -against. No guessing (contrast §6). It rides the existing lookup, grace window, conflict-collapse, and TTL — the -only new write is one key per body item. - -**Shape variance — parse defensively.** List-with-items → expand. `Status` / hollow / unparseable / absent body → -no items → no-op, degrade to §6. Never assume a body. - -### 5.1 Where the code goes - -A sibling to `RecordFact`, called from the same accept point -([audit_handler.go:258](../../internal/webhook/audit_handler.go#L258)): - -```go -// RecordDeleteCollectionFacts expands a deletecollection response body into one -// uid-only attribution fact per listed object, joined by UID against the per-object -// removal event. A no-op when the verb is not deletecollection or the body is -// absent/hollow/unparseable. Writes ONLY the uid-only key (no RV is supplied, so -// factKeyVariants yields just that variant). -func (a *AttributionIndex) RecordDeleteCollectionFacts(ctx context.Context, event auditv1.Event) error -``` - -`RecordFact` is unchanged (it already no-ops on the name-less collection event). The handler calls both; for a -`deletecollection`, only the expander does work. The single-object fast path is never branched. - -### 5.2 Reason code - -When a removal event matches an expander fact, `attributionResultForMatch` -([attribution_index.go:332](../../internal/queue/attribution_index.go#L332)) today returns `weak` (uid-only). Add -`AttributionExactDeleteCollectionItem` and return it when `fact.Verb == "deletecollection"`, so collection-member -attributions are distinct from generic weak matches in metrics — realizing the `exact-deletecollection-item` -reason code that merge-readiness §3.5 lists as unused. +The rest of this document still binds. **§2, the deletion-as-intent render rule, is untouched**, and +it is what keeps the collection window short: the removal is attributed at delete-REQUEST time, when +`deletionTimestamp` is set, so finalizers do not stretch it. §3's argument that a collection member +must join by UID rather than RV also still holds, and is why the uid tier sits above the scope tier. ## 6. The hard case — hollow / empty body (aggregated & metadata-only) @@ -217,9 +195,25 @@ objects**. The owner's trap: *in a few seconds you could see more than one `dele plumbing to carry a scope-cause into the commit-window builder. A deliberate fast-follow. - **Option C — commit as the explicit unresolved author, document the limit.** v1. -**v1: Option C.** The hard case is a narrow intersection (aggregated/hollow body ∧ supports `deletecollection` ∧ -watched ∧ attributed-author mode), and the failure is *degraded attribution*, not wrong state or a guessed author — which -is the correct conservative outcome. **Ship §2 + §5 now; Option B is the named fast-follow; Option A is rejected.** +**RESOLVED, and not by Option C.** The hollow-body case was shipped as **scope matching**, which is +Option A bounded until it is safe rather than rejected outright. Three things bound the +over-attribution the rejection was about, and the third is the one that changes the verdict: + +- **Namespace and selector narrow the scope.** The selector is the *intent the actor stated*, read + off the request URI, and it is present even when the body is not. An empty selector matches + everything of the type in the namespace, which is exactly what `--all` means. +- **Precedence keeps anything with its own fact out.** The scope tier is the weakest evidence the + join has, so it is reached only when every more specific tier missed. The unrelated + `kubectl delete configmap x` in the same window is claimed by its OWN fact and never reaches it. +- **The window is short**, because of §2. Under deletion-as-intent the removal happens at + delete-request time, so the window only has to cover audit batching plus clock skew — 30s by + default, against a fact TTL of ten minutes. That is what makes the scope match safe, and it is not + something the original framing, where attribution chased the eventual removal, could have offered. + +The result is the reverse of the old degradation: the aggregated and metadata-only cases that used +to ship committer-authored now resolve, and a production cluster with +`--audit-webhook-truncate-enabled` — the one MOST likely to send no body for a large collection +delete — is the one that gains most. Option B (`Co-authored-by`) is no longer needed for this case. ## 7. Recommendation at a glance @@ -227,22 +221,32 @@ is the correct conservative outcome. **Ship §2 + §5 now; Option B is the named |---|---|---| | Any delete (single or collection member) | **Remove file at intent time; never commit `deletionTimestamp`** (§2) | Git is intent; reversible invariant; manifests stay re-appliable. | | Finalizer object | **Removed immediately, attributed to the delete-requester**; finalizer cleanup is runtime no-op in Git | Reframe dissolves the old delay/conflict; controllers still finalize in-cluster. | -| Collection delete, body present | **Per-UID expander (§5)** credits the actor on each removal | API server states "these exact objects, by this user." | -| Collection delete, hollow body | **Explicit unresolved author (Option C, §6)** | Narrow; degraded attribution beats a wrong author. | +| Collection delete, body present | **One collection fact**, joined by **uid membership** (`collection_uid`) | API server states "these exact objects, by this user." | +| Collection delete, hollow body | **The same collection fact**, joined by **scope** — type, namespace, selector, window (`collection_scope`) | Bounded by precedence and a short window (§6); resolves what used to degrade. | | Stuck `Terminating` | **Operational status/metric** (§2.5), file already absent | Don't pollute intent with runtime state. | ## 8. Observability & diagnostics -- **`AttributionExactDeleteCollectionItem`** flows onto `AttributionResolutionsTotal{result=…}` via - `recordAttributionResolution` ([author_resolver.go:169](../../internal/watch/author_resolver.go#L169)) — a - dashboard can show collection-member precise attributions vs. unresolved outcomes. -- **Expander write counter** (`op="deletecollection_expanded"`) on `AttributionFactEventsTotal` via the existing - `recordFactEvent` hook ([attribution_index.go:433](../../internal/queue/attribution_index.go#L433)). -- **Stuck-finalizer / terminating diagnostics** (§2.5): surface long-`Terminating` watched objects whose files we - already removed, so logical absence never hides a stuck deletion. Optional **secondary diagnostic - attribution**: record *who* cleared the finalizer (the finalizer-clearing actor) as a diagnostic signal — a - metric label or debug log — **never** as the Git author (that stays the delete-requester, and the event is a - no-op commit anyway). v1 may ship the metric and defer the richer reporting. +**The `exact_deletecollection_item` result label is gone**, and so is the expander's +`op="deletecollection_expanded"` write counter. Two labels replace the one, because the match is now +two-tiered and the tiers carry different confidence: + +- **`collection_uid`** — the removal's uid was in the set the API server said it deleted. No + over-attribution risk at all. +- **`collection_scope`** — matched by namespace, selector, and window alone. Weaker evidence, and + the reason the window is short. + +Both flow onto `AttributionResolutionsTotal{result=…}`, so a dashboard can now separate *precise* +collection credit from *scoped* collection credit rather than seeing one bucket. +`attribution_collection_degraded_total{reason}` counts a collection fact published WITHOUT its uid +set, which is what turns the second tier from an inference into a measurement. See +[`interpreting-metrics.md`](../interpreting-metrics.md). + +**Stuck-finalizer / terminating diagnostics** (§2.5) are unchanged: surface long-`Terminating` +watched objects whose files we already removed, so logical absence never hides a stuck deletion. +Optional **secondary diagnostic attribution**: record *who* cleared the finalizer as a diagnostic +signal — a metric label or debug log — **never** as the Git author (that stays the delete-requester, +and the event is a no-op commit anyway). ## 9. Tests @@ -255,15 +259,20 @@ is the correct conservative outcome. **Ship §2 + §5 now; Option B is the named 3. A second Delete for an already-absent path diffs to **no-op** (covers finalizer-clear + eventual `DELETED` folding to nothing; assert no commit). May reuse existing writer no-op tests. -`internal/queue` (expander, §5): - -1. A list body with three items → three uid-only facts, each crediting the actor; **no** exact/rv-only keys. -2. A finalizer-pending item (`deletionTimestamp` + finalizers) **also** gets a fact crediting the actor (it is - *not* skipped). -3. A hollow / `Status` / unparseable / absent body → **no facts**, no error (degrade to §6). -4. A partial list writes facts only for items present (watch + sweep backstop the rest). -5. Join shape: a removal event with the item's UID and a *different* RV resolves to the actor via the uid-only - key (proves §3) and surfaces as `exact_deletecollection_item`. +`internal/queue` and `internal/watch` (the collection fact and its join, §5–§6): + +1. A collection delete publishes **one** fact carrying the actor, the namespace, the selector from the + request URI, and the uid set — never one fact per object. +2. A list body larger than the uid cap drops the set and counts + `attribution_collection_degraded_total{reason="uid_cap"}`; the fact still publishes. +3. A hollow / `Status` / unparseable / absent body still publishes a fact — with no uid set. This is the + case the expander produced nothing at all for. +4. Join shape, uid tier: a removal whose uid is in the set resolves to the actor as `collection_uid`, + even though its RV never matches (proves §3). +5. Join shape, scope tier: a removal with no uid set to consult resolves as `collection_scope` when the + namespace, selector and window cover it — and does NOT resolve when the selector rejects its labels, + when it is in another namespace, or when the window has passed. +6. Precedence: an object with its own fact never reaches either collection tier. ### 9.2 E2E — implemented @@ -287,8 +296,8 @@ these survive), never a global drop count, so they run against a reused cluster. 3. **`removes a single finalizer object at intent time too (the rule is not collection-specific)`.** A single named `Delete` of a finalizer-guarded configmap, as the actor: file removed at intent, authored by the actor, - object still `Terminating`; clearing the finalizer yields no further Git change. (Single deletes are attributed - by the existing `RecordFact`, not the expander — proving §2 is a general render rule.) + object still `Terminating`; clearing the finalizer yields no further Git change. (A single delete is + attributed by its own per-object fact, never by a collection one — proving §2 is a general render rule.) 4. **`scopes a label-selector collection delete to matching objects and leaves siblings`.** Two matching + one non-matching sibling; a label-selector collection delete removes only the matching files (authored by the @@ -299,14 +308,13 @@ these survive), never a global drop count, so they run against a reused cluster. - **§2 render rule:** `routeLiveTargetWatchEvent` reclassifies a `deletionTimestamp`-bearing event to Delete; no manifest ever carries `deletionTimestamp`/`deletionGracePeriodSeconds` (already true via sanitize); later finalizer/`DELETED` events fold to no-ops. Unit §9.1.1–9.1.3. -- **§5 expander:** additive (`RecordFact` unchanged); writes only the uid-only key per body item; finalizer items - attributed (not skipped); defensive parsing; `AttributionExactDeleteCollectionItem` + expander counter wired. - Unit §9.1.4–9.1.8. +- **§5 collection fact:** one fact per collection request, carrying scope, selector and (when the body + allowed it) the uid set; finalizer items attributed like any other, since §2 removes them at intent; + defensive parsing; `collection_uid` and `collection_scope` result labels wired. Unit §9.1.4–9.1.9. - **E2E §9.2.1–9.2.4** implemented, convergence-asserted; the finalizer showcase (§9.2.2) proves removal-at-intent with a *different* finalizer-clearing identity. -- **Hard case:** Option C documented in README/chart ("actor named when the API server returns the deleted set; - aggregated/hollow-body collection deletes are recorded as committer"); Option B noted as fast-follow; Option A - rejected so it isn't re-proposed. +- **Hard case:** resolved by scope matching (§6), bounded by namespace, selector, precedence and a short + window. Option B (`Co-authored-by`) is no longer needed for it. - **Reversibility honored:** main tree = resources intended to exist; richer `.deletions/` style records left as future enrichment, invariant intact. - Full validation per AGENTS.md: `task fmt → generate → manifests → vet → lint → test → test-e2e` (e2e sequential). diff --git a/internal/audit/outcome/outcome.go b/internal/audit/outcome/outcome.go index 7bf38ea2..6f2032b1 100644 --- a/internal/audit/outcome/outcome.go +++ b/internal/audit/outcome/outcome.go @@ -84,6 +84,17 @@ const ( // that carries no source-cluster annotation, so it names no ClusterProvider. Never credited to a // fallback; a rising rate means a producer is not stamping the annotation. MissingClusterAnnotation Outcome = "missing_cluster_annotation" + // NoAttributionFact — the event passed the accept gate and produced no attribution fact, so + // nothing was appended for it and no watch event can ever join it. It names an author nobody + // can look up: no user, or no resolvable object name on a non-collection verb — the shape an + // aggregated-API create has, since the API server assigns the name and the objectRef carries + // none. + // + // It is Dropped rather than Error: nothing failed, and the events are exactly the population + // that was previously counted queued (which claimed an append that was never owed), so this + // population was invisible. The handler is the only place it can be counted at all — the event + // is rejected before publication, so no fact-side counter ever sees it. + NoAttributionFact Outcome = "no_attribution_fact" // WriteError — a redis/enqueue failure; the event never reached the log. WriteError Outcome = "write_error" @@ -100,7 +111,7 @@ func (o Outcome) Category() Category { case NotNeeded, NilEvent, Stage, ReadOnlyOrUnknownVerb, FailedRequest, DryRun, UnchangedResourceVersion, MalformedAdditional, NonScaleSubresource, ShallowDropped, RVLessEmptyHighWater, OlderThanHighWater, NonNumericRV, - MissingClusterAnnotation: + MissingClusterAnnotation, NoAttributionFact: return Dropped case WriteError: return Error diff --git a/internal/audit/outcome/outcome_test.go b/internal/audit/outcome/outcome_test.go index f8d70e9c..eefeba9f 100644 --- a/internal/audit/outcome/outcome_test.go +++ b/internal/audit/outcome/outcome_test.go @@ -33,7 +33,10 @@ func TestOutcomeCategory(t *testing.T) { OlderThanHighWater: Dropped, NonNumericRV: Dropped, MissingClusterAnnotation: Dropped, - WriteError: Error, + // Accepted, and no fact was owed for it: dropped, never error, so the e2e invariant that + // gates on category="error" being zero is untouched by a type audit cannot identify. + NoAttributionFact: Dropped, + WriteError: Error, } for o, want := range cases { assert.Equalf(t, want, o.Category(), "category of %s", o) diff --git a/internal/controller/commitrequest_attribution_agreement_test.go b/internal/controller/commitrequest_attribution_agreement_test.go index 59f6bdfa..a7b3512b 100644 --- a/internal/controller/commitrequest_attribution_agreement_test.go +++ b/internal/controller/commitrequest_attribution_agreement_test.go @@ -5,6 +5,7 @@ package controller import ( "context" "testing" + "time" "github.com/go-logr/logr" "github.com/stretchr/testify/assert" @@ -22,13 +23,10 @@ import ( // "attribution is enabled but nothing matched within the grace". type absentLookup struct{} -func (absentLookup) LookupAuthorResolution( +func (absentLookup) Await( _ context.Context, - _ string, - _ schema.GroupVersionResource, - _ k8stypes.UID, - _ string, - _ bool, + _ queue.FactQuery, + _ time.Duration, ) queue.AuthorResolution { return queue.AuthorResolution{Result: queue.AttributionAbsent} } @@ -47,11 +45,13 @@ func windowOutcomeAttributionMissed(t *testing.T) git.AttributionOutcome { t.Helper() _, outcome := watch.NewAuthorResolver(absentLookup{}, 0, logr.Discard()).ResolveAuthor( context.Background(), - "default", - schema.GroupVersionResource{Version: "v1", Resource: "configmaps"}, - k8stypes.UID("uid-1"), - "101", - true, + watch.AuthorQuery{ + AuditRoute: "default", + GVR: schema.GroupVersionResource{Version: "v1", Resource: "configmaps"}, + UID: k8stypes.UID("uid-1"), + ResourceVersion: "101", + ExactCapable: true, + }, ) return outcome } diff --git a/internal/controller/gittarget_controller_test.go b/internal/controller/gittarget_controller_test.go index 2aa4f6b3..30a9d53b 100644 --- a/internal/controller/gittarget_controller_test.go +++ b/internal/controller/gittarget_controller_test.go @@ -1120,10 +1120,15 @@ var _ = Describe("GitTarget Controller Security", func() { // RequeueStreamSettleInterval — with the shared 10s `timeout` it equalled one, so a // deletion landing just after a reconcile lost the race by milliseconds. // - // Two intervals, not one: the previous budget covered exactly two ticks with no slack, so - // a CI runner busy enough to delay one of them by a second failed the spec on timing - // alone. Eventually returns as soon as the Secret is back, so the extra room is free on - // every run that was going to pass anyway. + // Three and a half intervals, not two: the two-interval budget was still exactly 30s and a + // CI runner busy enough to drop a tick failed the spec on timing alone — twice, months + // apart. Eventually returns as soon as the Secret is back, so the extra room is free on + // every run that was going to pass anyway; it is only ever spent by a run that was going + // to fail, and 15 seconds is a cheap price for not re-litigating a red build. + // + // It is deliberately still a BOUND rather than a generous number. The budget is this + // spec's implicit SLO — recreation must happen within about four ticks of the deletion — + // so a regression that made the requeue path slow rather than broken still fails here. Eventually(func(g Gomega) { var recreated corev1.Secret err := k8sClient.Get(ctx, secretKey, &recreated) @@ -1132,7 +1137,7 @@ var _ = Describe("GitTarget Controller Security", func() { g.Expect(ageKeyName).NotTo(BeEmpty()) g.Expect(string(ageKeyValue)).To(ContainSubstring("AGE-SECRET-KEY-")) g.Expect(recreated.Annotations).To(HaveKey(encryptionSecretRecipientAnnoKey)) - }, 2*RequeueStreamSettleInterval+timeout, interval).Should(Succeed()) + }, 7*RequeueStreamSettleInterval/2+timeout, interval).Should(Succeed()) Expect(k8sClient.Delete(ctx, target)).Should(Succeed()) Expect(k8sClient.Delete(ctx, gitProvider)).Should(Succeed()) diff --git a/internal/mutationlab/normalize/normalize.go b/internal/mutationlab/normalize/normalize.go index 1e4cbfee..9c67a7c6 100644 --- a/internal/mutationlab/normalize/normalize.go +++ b/internal/mutationlab/normalize/normalize.go @@ -155,6 +155,8 @@ type collector struct { cred *ordered rnd *ordered ns *ordered + // genName maps each full generated name to its stable generateName prefix. + genName map[string]string } func newCollector() *collector { @@ -168,6 +170,7 @@ func newCollector() *collector { node: newOrdered(), cred: newOrdered(), rnd: newOrdered(), + genName: map[string]string{}, ns: newOrdered(), } } @@ -209,6 +212,11 @@ func (c *collector) collectGenerateNameSuffix(m map[string]any) { return } c.rnd.add(name[len(gn):]) + // The FULL generated name is recorded too, because the suffix rule alone only reaches a `name` + // that sits beside its own `generateName`. An AdmissionReview carries the assigned name at + // request.name, a sibling of kind and namespace with no metadata around it, so without this the + // admission record churns the corpus on every capture (Row 18). + c.genName[name] = gn } func (c *collector) collectScalar(key string, v any) { @@ -298,6 +306,10 @@ type indices struct { cred map[string]string rnd map[string]string ns map[string]string + // genName maps a full generated name to its normalized form, and genNameByLen is that set + // longest-first for substring replacement. + genName map[string]string + genNameByLen []string // nsByLen / ipByLen / uidByLen are the namespace / IP / UID values sorted // longest-first, so substring replacement (in requestURIs and in managedFields // association keys like k:{"ip":"10.42.3.14"} or k:{"uid":""}) never @@ -334,6 +346,18 @@ func (c *collector) buildIndices() *indices { sort.SliceStable(idx.uidByLen, func(i, j int) bool { return len(idx.uidByLen[i]) > len(idx.uidByLen[j]) }) + idx.genName = map[string]string{} + for name, prefix := range c.genName { + if ph, ok := idx.rnd[name[len(prefix):]]; ok { + idx.genName[name] = prefix + ph + } + } + for name := range idx.genName { + idx.genNameByLen = append(idx.genNameByLen, name) + } + sort.SliceStable(idx.genNameByLen, func(i, j int) bool { + return len(idx.genNameByLen[i]) > len(idx.genNameByLen[j]) + }) return idx } @@ -366,6 +390,9 @@ func (idx *indices) transform(v any) any { case map[string]any: out := make(map[string]any, len(t)) for k, val := range t { + if isDroppedKey(k) { + continue + } // Values key off the original k (for type detection); the output key // is rewritten so a volatile value embedded in a managedFields // association key (k:{"ip":"10.42.3.14"}) does not churn the corpus. @@ -383,6 +410,22 @@ func (idx *indices) transform(v any) any { } } +// latencyAnnotationPrefix is the API server's per-request timing telemetry, which it attaches to an +// audit event's annotations only when a request was slow enough to be worth reporting. +const latencyAnnotationPrefix = "apiserver.latency.k8s.io/" + +// isDroppedKey reports whether a key is removed from the corpus rather than normalized. +// +// The latency annotations are dropped whole, not placeholder-ed, because BOTH halves of them are +// nondeterministic: the values are durations that differ every run, and the keys are present at all +// only when the request crossed the API server's slow-request threshold. Rewriting the values would +// still leave a corpus that churns between a fast run and a slow one, which is a diff that says +// nothing about behavior — and a corpus whose diff cannot be trusted to mean something is the one +// thing this tree must not become. +func isDroppedKey(key string) bool { + return strings.HasPrefix(key, latencyAnnotationPrefix) +} + // transformScalar rewrites a single key/value, falling back to a recursive // transform when the value is not a normalized leaf. func (idx *indices) transformScalar(key string, v any, parent map[string]any) any { @@ -390,6 +433,13 @@ func (idx *indices) transformScalar(key string, v any, parent map[string]any) an if rewritten, ok := idx.rewriteGeneratedName(v, parent); ok { return rewritten } + // The same name without its generateName sibling: an AdmissionReview carries the assigned + // name at request.name, beside kind and namespace rather than inside metadata. + if s, ok := stringVal(v); ok { + if ph, known := idx.genName[s]; known { + return ph + } + } } if key == "sourceIPs" || key == credentialIDKey { m := idx.ip @@ -426,13 +476,25 @@ func (idx *indices) transformStringScalar(key, s string) any { return idx.rewriteExpiredMessage(s) case key == "requestURI" || key == "selfLink": // The namespace appears embedded in the path; replace it as a substring so - // a unique per-run namespace does not churn the corpus. - return idx.replaceNamespaces(s) + // a unique per-run namespace does not churn the corpus. A generated name appears there too + // for any request that addresses such an object by name. + return idx.replaceGeneratedNames(idx.replaceNamespaces(s)) default: return s } } +// replaceGeneratedNames rewrites each full generated name, wherever it appears as a substring, to +// its normalized form — longest first, so a shorter name that prefixes a longer one cannot partially +// match. It is the substring counterpart of the metadata rule, for the places a generated name is +// embedded in a larger string rather than standing alone. +func (idx *indices) replaceGeneratedNames(s string) string { + for _, name := range idx.genNameByLen { + s = strings.ReplaceAll(s, name, idx.genName[name]) + } + return s +} + // rewriteExpiredMessage replaces the resourceVersions in a 410/Expired watch error // message with their placeholders, leaving every other message unchanged. func (idx *indices) rewriteExpiredMessage(s string) string { diff --git a/internal/mutationlab/normalize/normalize_test.go b/internal/mutationlab/normalize/normalize_test.go index 61ad149c..b2fffaa8 100644 --- a/internal/mutationlab/normalize/normalize_test.go +++ b/internal/mutationlab/normalize/normalize_test.go @@ -333,6 +333,47 @@ func TestNormalize_NumbersRoundTripAsIntegers(t *testing.T) { } } +// TestNormalize_LatencyAnnotationsAreDropped guards the corpus against a diff that means nothing. +// The API server attaches these only when a request was slow, so their values AND their presence +// vary run to run; one slow deletecollection is what put them in the tree in the first place. +func TestNormalize_LatencyAnnotationsAreDropped(t *testing.T) { + got := normJSON(t, `{"annotations":{`+ + `"apiserver.latency.k8s.io/total":"546.977655ms",`+ + `"apiserver.latency.k8s.io/etcd":"2.67214ms",`+ + `"authorization.k8s.io/decision":"allow"}}`) + want := `{"annotations":{"authorization.k8s.io/decision":"allow"}}` + if got[0] != want { + t.Errorf("\n got %s\nwant %s", got[0], want) + } +} + +// TestNormalize_GeneratedNameIsRewrittenWithoutItsGenerateNameSibling covers the shape that broke +// Row 18: an AdmissionReview carries the assigned name at request.name, beside kind and namespace +// rather than inside a metadata map, so the sibling rule cannot see a generateName there. The name +// is still the same per-run random string, and left alone it churns the corpus on every capture. +func TestNormalize_GeneratedNameIsRewrittenWithoutItsGenerateNameSibling(t *testing.T) { + got := normJSON(t, + `{"request":{"name":"cm-gen-x7k2p","object":{"metadata":{"generateName":"cm-gen-","name":"cm-gen-x7k2p"}}}}`) + want := `{"request":{"name":"cm-gen-","object":{"metadata":` + + `{"generateName":"cm-gen-","name":"cm-gen-"}}}}` + if got[0] != want { + t.Errorf("\n got %s\nwant %s", got[0], want) + } +} + +// TestNormalize_GeneratedNameIsRewrittenInsideARequestURI covers the other embedding: any request +// that addresses a generated-name object by name carries it in the path. +func TestNormalize_GeneratedNameIsRewrittenInsideARequestURI(t *testing.T) { + got := normJSON(t, + `{"requestURI":"/api/v1/namespaces/lab/configmaps/cm-gen-x7k2p",`+ + `"object":{"metadata":{"generateName":"cm-gen-","name":"cm-gen-x7k2p"}}}`) + want := `{"object":{"metadata":{"generateName":"cm-gen-","name":"cm-gen-"}},` + + `"requestURI":"/api/v1/namespaces/lab/configmaps/cm-gen-"}` + if got[0] != want { + t.Errorf("\n got %s\nwant %s", got[0], want) + } +} + func TestSingle(t *testing.T) { v, err := Single(json.RawMessage(`{"metadata":{"uid":"x"}}`)) if err != nil { diff --git a/internal/queue/attribution_index.go b/internal/queue/attribution_index.go deleted file mode 100644 index c5d7fe85..00000000 --- a/internal/queue/attribution_index.go +++ /dev/null @@ -1,592 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 - -package queue - -import ( - "context" - "encoding/json" - "fmt" - "strings" - "time" - - "github.com/redis/go-redis/v9" - "go.opentelemetry.io/otel/attribute" - "go.opentelemetry.io/otel/metric" - "k8s.io/apimachinery/pkg/runtime" - "k8s.io/apimachinery/pkg/runtime/schema" - "k8s.io/apimachinery/pkg/types" - auditv1 "k8s.io/apiserver/pkg/apis/audit/v1" - - "github.com/ConfigButler/gitops-reverser/internal/auditutil" - "github.com/ConfigButler/gitops-reverser/internal/telemetry" -) - -// DefaultAttributionFactTTL is how long an attribution fact is retained in Redis -// waiting for the matching watch event to join it. Facts are never object state, so -// they expire on their own — nothing deletes them. After it elapses a miss is simply -// "absent": the v3 schema keeps no tombstone, so an aged-out fact is indistinguishable -// from one that never arrived. Configurable via --author-attribution-ttl. -const DefaultAttributionFactTTL = 10 * time.Minute - -// DefaultKeyPrefix is the root namespace every Redis key (cursors, facts, and command -// author records alike) carries when --redis-key-prefix is not set. It is also the value -// every release before the flag existed used, so the default is a no-op upgrade. -const DefaultKeyPrefix = "gitops-reverser" - -const ( - // attributionKeySuffix namespaces audit-sourced resource author facts under the - // top-level author domain, e.g. - // "gitops-reverser:author:v1:audit:route:::...". - attributionKeySuffix = ":author:v1:audit:" - // routeKeyInfix carries the AUDIT ROUTE dimension so a fact from cluster A never joins a watch - // event from cluster B — the rv-only hatch especially, since RV is not globally unique. The - // route is what the audit events arrived under, NOT the ClusterProvider's name: an API server - // has one webhook backend and posts under one route, so several providers naming one cluster - // all declare that route and share its facts (ClusterProvider.AuditRoute()). It is spelled - // "route" rather than "auditRoute" because the key already says audit one segment earlier. It - // sits right after attributionKeySuffix so a route's facts share a single glob prefix - // "…:author:v1:audit:route::*", so one route's facts are inspectable with a single SCAN. - routeKeyInfix = "route:" - // factObjectInfix groups every fact for one object under one prefix, so a SCAN of - // ":object::*" shows the whole history-in-flight for that object. - factObjectInfix = ":object:" - // factRVInfix is the type-scoped rv-only escape hatch, a sibling of object: for a - // fact that has an RV but no UID (§5 of redis-key-schema-v3.md). - factRVInfix = ":rv:" - // factLastLeaf is the latest-writer-wins pointer for an object, written on every - // update and consulted only when the immutable exact key misses. - factLastLeaf = "last" - - attributionFactScanBatchSize = 100 - serviceAccountUserPrefix = "system:serviceaccount:" -) - -// AttributionResult is the bounded resolver outcome recorded for each watch event. -type AttributionResult string - -const ( - // AttributionExactUser is an exact UID+resourceVersion match for a human user. - AttributionExactUser AttributionResult = "exact_user" - // AttributionExactServiceAccount is an exact UID+resourceVersion match for a named service account. - AttributionExactServiceAccount AttributionResult = "exact_serviceaccount" - // AttributionWeak is a non-exact match: the uid-latest :last pointer or the rv-only - // escape hatch, used by known RV-mismatch events and no-UID facts respectively. - AttributionWeak AttributionResult = "weak" - // AttributionExactDeleteCollectionItem is a match to a fact expanded from a - // deletecollection response body — a precise per-object credit for one member of a - // collection delete, joined by UID via :last (the body item's RV is the pre-delete - // RV and never matches the removal event's RV). The reason is driven by the value's - // verb, not by which key matched. - AttributionExactDeleteCollectionItem AttributionResult = "exact_deletecollection_item" - // AttributionCollectionUID is a removal matched to a deletecollection fact whose uid set - // contains this object. There is no over-attribution risk in it: either the API server said it - // deleted this object, or it did not. - AttributionCollectionUID AttributionResult = "collection_uid" - // AttributionCollectionScope is a removal matched to a deletecollection fact by scope alone — - // same type and namespace, selector accepting the object's labels, within the collection window. - // It is the weakest evidence the join has, which is why it is reached only when every more - // specific tier missed. - AttributionCollectionScope AttributionResult = "collection_scope" - // AttributionAbsent means no usable author fact matched before the grace elapsed. - AttributionAbsent AttributionResult = "absent" -) - -// AuthorFact is the minimal attribution fact stored per accepted, mutating audit -// event and read back by the watch-event resolver. It names an author candidate and -// carries the evidence needed to decide confidence; it is never object state. v3 moves -// the object identity (group-resource, namespace, name, uid) off the key and into the -// value, so the fact is self-describing. -type AuthorFact struct { - GroupResource string `json:"groupResource,omitempty"` - Namespace string `json:"namespace,omitempty"` - Name string `json:"name,omitempty"` - UID string `json:"uid,omitempty"` - Author string `json:"author"` - DisplayName string `json:"displayName,omitempty"` - Email string `json:"email,omitempty"` - Verb string `json:"verb,omitempty"` - Subresource string `json:"subresource,omitempty"` - AuditID string `json:"auditID,omitempty"` - ResourceVersion string `json:"resourceVersion,omitempty"` - StageTimestamp string `json:"stageTimestamp,omitempty"` - IsServiceAccount bool `json:"isServiceAccount,omitempty"` - - // LabelSelector is the selector the request URI expressed, carried on a COLLECTION fact only. - // It is the intent the actor stated, and evaluating it against the object a watch event carries - // is a better test of membership than reading back a list the API server may not have sent. - // Empty means the collection covered everything of its type in its namespace, which is what - // --all means. - LabelSelector string `json:"labelSelector,omitempty"` - // UIDs is the set of objects a collection delete covered, reduced from the response body at the - // receiver, on a COLLECTION fact only. It is absent when the API server sent no body — a - // truncated, aggregated, or metadata-only response — and when the set was larger than the cap, - // in which case the join falls back to scope matching, which is already correct. - UIDs []string `json:"uids,omitempty"` -} - -// AuthorResolution is the structured result of an attribution lookup. -type AuthorResolution struct { - Fact AuthorFact - Result AttributionResult -} - -// AttributionIndex is the optional Redis-backed lookup table that names a commit -// author from audit facts. It is built from a RedisStore (sharing its connection) only -// when author attribution is enabled, and stores only attribution facts keyed for a -// join against watch events — never object state, and never the resume cursors (those -// belong to RedisStore, which is required regardless of this index). -type AttributionIndex struct { - client *redis.Client - factTTL time.Duration - // keyPrefix is the root namespace this index writes and reads under, shared with the - // RedisStore that built it. Empty only in tests constructed by hand; every key builder - // resolves it so an empty value still lands on DefaultKeyPrefix. - keyPrefix string -} - -// RecordFact stores the attribution fact for one accepted, mutating audit event. A -// UID-bearing fact writes the immutable exact key (uid+rv) and overwrites the :last -// pointer; a fact that has an RV but no UID writes the type-scoped rv-only key instead. -// It is a no-op for events without an objectRef, a resolvable name, or a user — those -// can never name an author. The caller (the audit handler) has already rejected reads, -// failures, dry-runs, and non-ResponseComplete stages. -func (a *AttributionIndex) RecordFact(ctx context.Context, auditRoute string, event auditv1.Event) error { - if event.ObjectRef == nil { - return nil - } - group := event.ObjectRef.APIGroup - resource := event.ObjectRef.Resource - if resource == "" { - return nil - } - - // A deletecollection is name-less, so it can never name a single object. When the - // API server returns the deleted set, expand it into one fact per object instead. - if strings.EqualFold(event.Verb, "deletecollection") { - return a.RecordDeleteCollectionFacts(ctx, auditRoute, event) - } - - op, _ := auditutil.VerbToOperation(event.Verb) - identity := auditutil.IdentityFromAuditEvent(event, op) - if identity.Name == "" { - return nil - } - - user := resolveUserInfo(event) - if user.Username == "" { - return nil - } - - gr := groupResourceKey(group, resource) - rv := resourceVersionFromEvent(event) - uid := string(identity.UID) - fact := AuthorFact{ - GroupResource: gr, - Namespace: identity.Namespace, - Name: identity.Name, - UID: uid, - Author: user.Username, - DisplayName: user.DisplayName, - Email: user.Email, - Verb: event.Verb, - Subresource: event.ObjectRef.Subresource, - AuditID: string(event.AuditID), - ResourceVersion: rv, - IsServiceAccount: strings.HasPrefix(user.Username, serviceAccountUserPrefix), - } - if !event.StageTimestamp.IsZero() { - fact.StageTimestamp = event.StageTimestamp.UTC().Format(time.RFC3339Nano) - } - - raw, err := json.Marshal(fact) - if err != nil { - return fmt.Errorf("marshal attribution fact: %w", err) - } - - wrote, err := a.writeFactKeys(ctx, auditRoute, gr, uid, rv, raw) - if err != nil { - return err - } - if wrote { - a.recordFactEvent(ctx, "written") - a.recordFactIndexSize(ctx) - } - return nil -} - -// writeFactKeys persists a single-object fact under the keys it can compute: the -// immutable exact key (uid+rv) plus the last-writer-wins :last pointer when a UID is -// known, or the type-scoped rv-only escape hatch when the fact has an RV but no UID. -// The exact key is written once per (uid, rv) and never contended, so there is no -// conflict marking. It reports whether any key was written. -func (a *AttributionIndex) writeFactKeys( - ctx context.Context, - auditRoute, gr, uid, rv string, - raw []byte, -) (bool, error) { - switch { - case uid != "": - if rv != "" { - if err := a.setFact(ctx, a.factKeyExact(auditRoute, gr, uid, rv), raw); err != nil { - return false, fmt.Errorf("store exact attribution fact: %w", err) - } - } - if err := a.setFact(ctx, a.factKeyLast(auditRoute, gr, uid), raw); err != nil { - return false, fmt.Errorf("store last attribution fact: %w", err) - } - return true, nil - case rv != "": - // The §5 escape hatch: a UID-bearing fact's rv-only key would be dead (the watch - // side always carries a UID and resolves via object::… first), so it is - // written only when there is no UID. - if err := a.setFact(ctx, a.factKeyRV(auditRoute, gr, rv), raw); err != nil { - return false, fmt.Errorf("store rv-only attribution fact: %w", err) - } - return true, nil - default: - return false, nil - } -} - -// RecordDeleteCollectionFacts expands a deletecollection response body into one -// uid-latest (:last) attribution fact per listed object, joined by UID against the -// per-object removal watch event. It is a no-op for any other verb, or when the body is -// absent, hollow, or unparseable — an aggregated / metadata-only deletecollection then -// degrades to a committer-authored removal. -// -// It writes ONLY the :last key: the body item's resourceVersion is the pre-delete RV, -// which no watch removal event ever presents, so the exact and rv-only keys would be -// dead. Finalizer-pending items are NOT skipped — under the deletion-as-intent rule a -// deletionTimestamp already removes the file, so the actor who ran the collection delete -// is credited with that removal even while Kubernetes finalization is still in flight. -// See docs/spec/deletecollection-attribution-expander.md. -func (a *AttributionIndex) RecordDeleteCollectionFacts( - ctx context.Context, - auditRoute string, - event auditv1.Event, -) error { - if !strings.EqualFold(event.Verb, "deletecollection") || event.ObjectRef == nil || event.ObjectRef.Resource == "" { - return nil - } - user := resolveUserInfo(event) - if user.Username == "" { - return nil - } - items := deleteCollectionItems(event.ResponseObject) - if len(items) == 0 { - return nil - } - - base := AuthorFact{ - Author: user.Username, - DisplayName: user.DisplayName, - Email: user.Email, - Verb: "deletecollection", - AuditID: string(event.AuditID), - IsServiceAccount: strings.HasPrefix(user.Username, serviceAccountUserPrefix), - } - if !event.StageTimestamp.IsZero() { - base.StageTimestamp = event.StageTimestamp.UTC().Format(time.RFC3339Nano) - } - return a.storeDeleteCollectionFacts( - ctx, - auditRoute, - event.ObjectRef.APIGroup, - event.ObjectRef.Resource, - items, - base, - ) -} - -// storeDeleteCollectionFacts writes one :last fact per joinable item, carrying the -// per-item object identity in the value, and records the expander metrics when at least -// one item was written. -func (a *AttributionIndex) storeDeleteCollectionFacts( - ctx context.Context, - auditRoute string, - group, resource string, - items []deleteCollectionItem, - base AuthorFact, -) error { - gr := groupResourceKey(group, resource) - base.GroupResource = gr - expanded := false - for _, item := range items { - if item.Name == "" || item.UID == "" { - continue - } - fact := base - fact.Namespace = item.Namespace - fact.Name = item.Name - fact.UID = string(item.UID) - raw, err := json.Marshal(fact) - if err != nil { - return fmt.Errorf("marshal deletecollection fact: %w", err) - } - if err := a.setFact(ctx, a.factKeyLast(auditRoute, gr, string(item.UID)), raw); err != nil { - return fmt.Errorf("store deletecollection fact %q: %w", item.Name, err) - } - expanded = true - } - if expanded { - a.recordFactEvent(ctx, "deletecollection_expanded") - a.recordFactIndexSize(ctx) - } - return nil -} - -// deleteCollectionItem is the per-object identity read from a deletecollection -// response list. -type deleteCollectionItem struct { - Namespace string - Name string - UID types.UID -} - -// deleteCollectionItems parses the per-object identities from a deletecollection -// response body. It accepts any list-shaped body (a typed "…List", a v1.List, or -// anything carrying an "items" array) and returns nil for a Status, hollow, or -// unparseable body — the caller then degrades to a committer-authored removal. -func deleteCollectionItems(obj *runtime.Unknown) []deleteCollectionItem { - if obj == nil || len(obj.Raw) == 0 { - return nil - } - var envelope struct { - Items []struct { - Metadata struct { - Namespace string `json:"namespace"` - Name string `json:"name"` - UID types.UID `json:"uid"` - } `json:"metadata"` - } `json:"items"` - } - if err := json.Unmarshal(obj.Raw, &envelope); err != nil { - return nil - } - items := make([]deleteCollectionItem, 0, len(envelope.Items)) - for _, it := range envelope.Items { - items = append(items, deleteCollectionItem{ - Namespace: it.Metadata.Namespace, - Name: it.Metadata.Name, - UID: it.Metadata.UID, - }) - } - return items -} - -// LookupAuthor finds the strongest attribution fact for a watch event. ok=false means -// no fact matched (yet) — the caller ships as committer. exactCapable selects the join -// policy: see LookupAuthorResolution. -func (a *AttributionIndex) LookupAuthor( - ctx context.Context, - auditRoute string, - gvr schema.GroupVersionResource, - uid types.UID, - rv string, - exactCapable bool, -) (AuthorFact, bool) { - resolution := a.LookupAuthorResolution(ctx, auditRoute, gvr, uid, rv, exactCapable) - return resolution.Fact, resolution.Result != AttributionAbsent -} - -// LookupAuthorResolution finds the strongest attribution fact and classifies the match. -// It is event-kind-aware: -// -// - An exact-capable event (ADDED / MODIFIED) tries only the immutable exact key -// object:: and the rv-only escape hatch; it never falls through to the -// last-writer-wins :last pointer, because that pointer may name a different, older -// author than the create/update this event represents. -// - A known RV-mismatch event (DELETED, deletecollection-expanded removal) additionally -// consults object::last, whose RV deliberately never matches. -// -// A miss returns AttributionAbsent; there is no tombstone and so no expired outcome. -func (a *AttributionIndex) LookupAuthorResolution( - ctx context.Context, - auditRoute string, - gvr schema.GroupVersionResource, - uid types.UID, - rv string, - exactCapable bool, -) AuthorResolution { - gr := groupResourceKey(gvr.Group, gvr.Resource) - if uid != "" && rv != "" { - if res, ok := a.matchFactKey(ctx, a.factKeyExact(auditRoute, gr, string(uid), rv), false); ok { - return res - } - } - if !exactCapable && uid != "" { - if res, ok := a.matchFactKey(ctx, a.factKeyLast(auditRoute, gr, string(uid)), true); ok { - return res - } - } - if rv != "" { - if res, ok := a.matchFactKey(ctx, a.factKeyRV(auditRoute, gr, rv), true); ok { - return res - } - } - return AuthorResolution{Result: AttributionAbsent} -} - -// matchFactKey reads one candidate key and turns a present, author-bearing fact into a -// resolution. weak marks a non-exact match (the :last or rv-only key). -func (a *AttributionIndex) matchFactKey(ctx context.Context, key string, weak bool) (AuthorResolution, bool) { - raw, err := a.client.Get(ctx, key).Bytes() - if err != nil { - return AuthorResolution{}, false - } - var fact AuthorFact - if err := json.Unmarshal(raw, &fact); err != nil || fact.Author == "" { - return AuthorResolution{}, false - } - a.recordFactEvent(ctx, "matched") - return AuthorResolution{Fact: fact, Result: attributionResultForFact(fact, weak)}, true -} - -// attributionResultForFact derives the reason from the matched fact and whether the -// match was weak. A deletecollection fact is precise per-object credit regardless of -// which key matched, so its verb (read from the value) wins. -func attributionResultForFact(fact AuthorFact, weak bool) AttributionResult { - if strings.EqualFold(fact.Verb, "deletecollection") { - return AttributionExactDeleteCollectionItem - } - if weak { - return AttributionWeak - } - if fact.IsServiceAccount { - return AttributionExactServiceAccount - } - return AttributionExactUser -} - -// routeFactPrefix is the per-route glob prefix under which every fact for one audit route lives, -// e.g. "gitops-reverser:author:v1:audit:route:prod-eu-1:". -func (a *AttributionIndex) routeFactPrefix(auditRoute string) string { - return resolveKeyPrefix(a.keyPrefix) + attributionKeySuffix + routeKeyInfix + escapeKeyField(auditRoute) + ":" -} - -// factKeyBase is the per-(cluster,type) prefix shared by every fact key, e.g. -// "gitops-reverser:author:v1:audit:route:default:apps/deployments". -func (a *AttributionIndex) factKeyBase(auditRoute, gr string) string { - return a.routeFactPrefix(auditRoute) + gr -} - -// factKeyExact is the immutable per-write fact key, e.g. -// "gitops-reverser:author:v1:audit:route:default:apps/deployments:object::101". -func (a *AttributionIndex) factKeyExact(auditRoute, gr, uid, rv string) string { - return a.factKeyBase(auditRoute, gr) + factObjectInfix + escapeKeyField(uid) + ":" + escapeKeyField(rv) -} - -// factKeyLast is the latest-writer-wins pointer for an object, e.g. -// "gitops-reverser:author:v1:audit:route:default:apps/deployments:object::last". -func (a *AttributionIndex) factKeyLast(auditRoute, gr, uid string) string { - return a.factKeyBase(auditRoute, gr) + factObjectInfix + escapeKeyField(uid) + ":" + factLastLeaf -} - -// factKeyRV is the (cluster, type)-scoped rv-only escape hatch, e.g. -// "gitops-reverser:author:v1:audit:route:default:apps/deployments:rv:101". RV is opaque per the -// Kubernetes API contract and not globally unique — not even within a cluster's type, and -// certainly not across clusters — so this key always includes both the cluster and the type. -func (a *AttributionIndex) factKeyRV(auditRoute, gr, rv string) string { - return a.factKeyBase(auditRoute, gr) + factRVInfix + escapeKeyField(rv) -} - -// setFact writes one fact value under its key with the bounded fact TTL. No sibling -// keys: v3 keeps no :seen tombstone and no :miss marker. -func (a *AttributionIndex) setFact(ctx context.Context, key string, raw []byte) error { - return a.client.Set(ctx, key, raw, a.factTTL).Err() -} - -func (a *AttributionIndex) recordFactEvent(ctx context.Context, op string) { - if telemetry.AttributionFactEventsTotal == nil { - return - } - telemetry.AttributionFactEventsTotal.Add(ctx, 1, metric.WithAttributes(attribute.String("op", op))) -} - -func (a *AttributionIndex) recordFactIndexSize(ctx context.Context) { - if telemetry.AttributionFactIndexSize == nil { - return - } - var cursor uint64 - var count int64 - pattern := resolveKeyPrefix(a.keyPrefix) + attributionKeySuffix + "*" - for { - keys, next, err := a.client.Scan(ctx, cursor, pattern, attributionFactScanBatchSize).Result() - if err != nil { - return - } - count += int64(len(keys)) - cursor = next - if cursor == 0 { - break - } - } - telemetry.AttributionFactIndexSize.Record(ctx, count) -} - -// groupResourceKey renders a GroupResource as an API-path-style segment: "configmaps" -// for the core group, "apps/deployments" otherwise. Write side and read side share it so -// the key never drifts. "/" never appears in a group or resource name, so the form stays -// unambiguously splittable — unlike schema.GroupResource.String()'s reversed dot form -// ("deployments.apps"), whose dot also collides with dotted group names. -func groupResourceKey(group, resource string) string { - if group == "" { - return resource - } - return group + "/" + resource -} - -// escapeKeyField neutralizes the ":" delimiter and the "%" escape character within a -// single key field. Group/resource and a UUID never contain either, so this is defensive -// for the uid/rv/namespace fields against a stray delimiter; everything else passes -// through unchanged for readability. Keys are only ever matched exactly, never parsed -// back, so escaping is one-way. -func escapeKeyField(s string) string { - if !strings.ContainsAny(s, "%:") { - return s - } - var b strings.Builder - b.Grow(len(s)) - for i := range len(s) { - switch s[i] { - case '%': - b.WriteString("%25") - case ':': - b.WriteString("%3A") - default: - b.WriteByte(s[i]) - } - } - return b.String() -} - -// resourceVersionFromEvent returns the event's ResourceVersion when one is available, -// or "" when it is not (deletes, collection verbs, shallow bodies). The post-write RV -// lives in the response object's metadata.resourceVersion; requestObject.resourceVersion -// is the pre-write RV on update-style requests, so it is intentionally ignored. -// objectRef.resourceVersion is usually the empty precondition RV on writes, so it is only -// the last resort. -func resourceVersionFromEvent(event auditv1.Event) string { - if rv := rvFromRawObject(event.ResponseObject); rv != "" { - return rv - } - if event.ObjectRef != nil { - return event.ObjectRef.ResourceVersion - } - return "" -} - -func rvFromRawObject(obj *runtime.Unknown) string { - if obj == nil || len(obj.Raw) == 0 { - return "" - } - var probe struct { - Metadata struct { - ResourceVersion string `json:"resourceVersion"` - } `json:"metadata"` - } - if err := json.Unmarshal(obj.Raw, &probe); err != nil { - return "" - } - return probe.Metadata.ResourceVersion -} diff --git a/internal/queue/attribution_index_deletecollection_test.go b/internal/queue/attribution_index_deletecollection_test.go deleted file mode 100644 index b7f792f0..00000000 --- a/internal/queue/attribution_index_deletecollection_test.go +++ /dev/null @@ -1,186 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 - -package queue - -import ( - "context" - "encoding/json" - "testing" - "time" - - "github.com/stretchr/testify/require" - authnv1 "k8s.io/api/authentication/v1" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/apimachinery/pkg/runtime" - "k8s.io/apimachinery/pkg/runtime/schema" - k8stypes "k8s.io/apimachinery/pkg/types" - auditv1 "k8s.io/apiserver/pkg/apis/audit/v1" -) - -func coreConfigmapsGVR() schema.GroupVersionResource { - return schema.GroupVersionResource{Group: "", Version: "v1", Resource: "configmaps"} -} - -type dcItem struct { - namespace, name, uid string - terminating bool -} - -// deleteCollectionEvent builds a core/configmaps deletecollection in namespace team-a -// authored by username, whose responseObject lists the given items as a ConfigMapList. -func deleteCollectionEvent(username string, items ...dcItem) auditv1.Event { - listItems := make([]interface{}, 0, len(items)) - for _, it := range items { - meta := map[string]interface{}{"namespace": it.namespace, "name": it.name, "uid": it.uid} - if it.terminating { - meta["deletionTimestamp"] = "2026-06-28T00:00:00Z" - meta["finalizers"] = []string{"example.com/cleanup"} - } - listItems = append(listItems, map[string]interface{}{"metadata": meta}) - } - body, _ := json.Marshal(map[string]interface{}{ - "apiVersion": "v1", "kind": "ConfigMapList", "items": listItems, - }) - return auditv1.Event{ - AuditID: "audit-dc", - Verb: "deletecollection", - Stage: auditv1.StageResponseComplete, - StageTimestamp: metav1.MicroTime{Time: time.Now()}, - User: authnv1.UserInfo{Username: username}, - ObjectRef: &auditv1.ObjectReference{ - APIVersion: "v1", - Resource: "configmaps", - Namespace: "team-a", - }, - ResponseObject: &runtime.Unknown{Raw: body}, - } -} - -// resolveDC looks up a removal event for one collection member at a later deletion RV, -// proving the join is by UID (the body item carried no RV at all) via the :last pointer. -// A collection removal is a known RV-mismatch event, so it is not exact-capable. -func resolveDC(ctx context.Context, idx *AttributionIndex, _, uid string) AuthorResolution { - return idx.LookupAuthorResolution(ctx, "default", coreConfigmapsGVR(), k8stypes.UID(uid), "9999", false) -} - -func TestRecordDeleteCollectionFacts_ExpandsListToPerObjectFacts(t *testing.T) { - idx := newTestAttributionIndex(t) - ctx := context.Background() - - require.NoError(t, idx.RecordFact(ctx, "default", deleteCollectionEvent("alice", - dcItem{namespace: "team-a", name: "a", uid: "uid-a"}, - dcItem{namespace: "team-a", name: "b", uid: "uid-b"}, - dcItem{namespace: "team-a", name: "c", uid: "uid-c"}, - ))) - - for _, it := range []struct{ name, uid string }{{"a", "uid-a"}, {"b", "uid-b"}, {"c", "uid-c"}} { - res := resolveDC(ctx, idx, it.name, it.uid) - require.Equal(t, AttributionExactDeleteCollectionItem, res.Result, it.name) - require.Equal(t, "alice", res.Fact.Author, it.name) - } -} - -// TestRecordDeleteCollectionFacts_FinalizerItemAttributed proves the v1 choice: a -// finalizer-pending member (deletionTimestamp set) is NOT skipped — the actor who ran -// the collection delete is credited with its removal-at-intent. -func TestRecordDeleteCollectionFacts_FinalizerItemAttributed(t *testing.T) { - idx := newTestAttributionIndex(t) - ctx := context.Background() - - require.NoError(t, idx.RecordFact(ctx, "default", deleteCollectionEvent("alice", - dcItem{namespace: "team-a", name: "plain", uid: "uid-plain"}, - dcItem{namespace: "team-a", name: "stuck", uid: "uid-stuck", terminating: true}, - ))) - - for _, it := range []struct{ name, uid string }{{"plain", "uid-plain"}, {"stuck", "uid-stuck"}} { - res := resolveDC(ctx, idx, it.name, it.uid) - require.Equal(t, AttributionExactDeleteCollectionItem, res.Result, it.name) - require.Equal(t, "alice", res.Fact.Author, it.name) - } -} - -// TestRecordDeleteCollectionFacts_HollowBodyWritesNothing covers the hard case: an -// aggregated / metadata-only / unparseable / absent body yields no facts and no error, -// degrading to a committer-authored removal. -func TestRecordDeleteCollectionFacts_HollowBodyWritesNothing(t *testing.T) { - idx := newTestAttributionIndex(t) - ctx := context.Background() - - statusEvent := deleteCollectionEvent("alice") - statusEvent.ResponseObject = &runtime.Unknown{Raw: []byte(`{"kind":"Status","status":"Success"}`)} - require.NoError(t, idx.RecordFact(ctx, "default", statusEvent)) - - absentEvent := deleteCollectionEvent("alice") - absentEvent.ResponseObject = nil - require.NoError(t, idx.RecordFact(ctx, "default", absentEvent)) - - badEvent := deleteCollectionEvent("alice") - badEvent.ResponseObject = &runtime.Unknown{Raw: []byte(`{not json`)} - require.NoError(t, idx.RecordFact(ctx, "default", badEvent)) - - require.Equal(t, AttributionAbsent, resolveDC(ctx, idx, "anything", "uid-x").Result) -} - -// TestRecordDeleteCollectionFacts_PartialListOnlyWritesPresent proves a partial body -// (a large collection delete that returned only some items) attributes what it has and -// leaves the rest without a usable fact; live attributed writes use the explicit -// unresolved author for those events. -func TestRecordDeleteCollectionFacts_PartialListOnlyWritesPresent(t *testing.T) { - idx := newTestAttributionIndex(t) - ctx := context.Background() - - require.NoError(t, idx.RecordFact(ctx, "default", deleteCollectionEvent("alice", - dcItem{namespace: "team-a", name: "a", uid: "uid-a"}, - dcItem{namespace: "team-a", name: "b", uid: "uid-b"}, - ))) - - require.Equal(t, AttributionExactDeleteCollectionItem, resolveDC(ctx, idx, "a", "uid-a").Result) - require.Equal(t, AttributionExactDeleteCollectionItem, resolveDC(ctx, idx, "b", "uid-b").Result) - require.Equal(t, AttributionAbsent, resolveDC(ctx, idx, "c", "uid-c").Result, - "an item absent from the body gets no fact") -} - -// TestRecordDeleteCollectionFacts_SkipsItemsMissingUIDOrName guards the per-item loop: -// a body item without a usable UID or name cannot be joined and is skipped silently. -func TestRecordDeleteCollectionFacts_SkipsItemsMissingUIDOrName(t *testing.T) { - idx := newTestAttributionIndex(t) - ctx := context.Background() - - require.NoError(t, idx.RecordFact(ctx, "default", deleteCollectionEvent("alice", - dcItem{namespace: "team-a", name: "good", uid: "uid-good"}, - dcItem{namespace: "team-a", name: "", uid: "uid-noname"}, - dcItem{namespace: "team-a", name: "nouid", uid: ""}, - ))) - - require.Equal(t, AttributionExactDeleteCollectionItem, resolveDC(ctx, idx, "good", "uid-good").Result) -} - -// TestRecordDeleteCollectionFacts_ServiceAccountActor confirms a service-account actor -// is credited by its own username and flagged, matching the single-object path. -func TestRecordDeleteCollectionFacts_ServiceAccountActor(t *testing.T) { - idx := newTestAttributionIndex(t) - ctx := context.Background() - const sa = "system:serviceaccount:flux-system:kustomize-controller" - - require.NoError(t, idx.RecordFact(ctx, "default", deleteCollectionEvent(sa, - dcItem{namespace: "team-a", name: "a", uid: "uid-a"}, - ))) - - res := resolveDC(ctx, idx, "a", "uid-a") - require.Equal(t, AttributionExactDeleteCollectionItem, res.Result) - require.Equal(t, sa, res.Fact.Author) - require.True(t, res.Fact.IsServiceAccount) -} - -// TestRecordDeleteCollectionFacts_NonDeleteCollectionVerbIsNoOp guards the verb gate. -func TestRecordDeleteCollectionFacts_NonDeleteCollectionVerbIsNoOp(t *testing.T) { - idx := newTestAttributionIndex(t) - ctx := context.Background() - - require.NoError( - t, - idx.RecordDeleteCollectionFacts(ctx, "default", mutationEvent("delete", "uid-1", "101", "alice")), - ) - require.Equal(t, AttributionAbsent, - idx.LookupAuthorResolution(ctx, "default", appsDeploymentGVR(), "uid-1", "101", true).Result) -} diff --git a/internal/queue/attribution_index_test.go b/internal/queue/attribution_index_test.go deleted file mode 100644 index ddab34c8..00000000 --- a/internal/queue/attribution_index_test.go +++ /dev/null @@ -1,700 +0,0 @@ -// SPDX-License-Identifier: Apache-2.0 - -package queue - -import ( - "context" - "fmt" - "testing" - "time" - - "github.com/alicebob/miniredis/v2" - "github.com/stretchr/testify/require" - authnv1 "k8s.io/api/authentication/v1" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - "k8s.io/apimachinery/pkg/runtime" - "k8s.io/apimachinery/pkg/runtime/schema" - k8stypes "k8s.io/apimachinery/pkg/types" - auditv1 "k8s.io/apiserver/pkg/apis/audit/v1" - - "github.com/ConfigButler/gitops-reverser/internal/telemetry" -) - -func newTestRedisStore(t *testing.T) *RedisStore { - t.Helper() - store, _ := newTestRedisStoreWithRedis(t) - return store -} - -func newTestRedisStoreWithRedis(t *testing.T) (*RedisStore, *miniredis.Miniredis) { - t.Helper() - mr := miniredis.RunT(t) - store, err := NewRedisStore(RedisStoreConfig{Addr: mr.Addr()}) - require.NoError(t, err) - return store, mr -} - -func newTestAttributionIndex(t *testing.T) *AttributionIndex { - t.Helper() - idx, _ := newTestAttributionIndexWithRedis(t) - return idx -} - -func newTestAttributionIndexWithRedis(t *testing.T) (*AttributionIndex, *miniredis.Miniredis) { - t.Helper() - store, mr := newTestRedisStoreWithRedis(t) - return store.AttributionIndex(0), mr -} - -// mutationEvent builds an apps/deployments event for team-a/web authored by username, -// whose objectRef + responseObject carry uid and resourceVersion rv. -func mutationEvent(verb, uid, rv, username string) auditv1.Event { - const namespace, name = "team-a", "web" - body := fmt.Sprintf(`{"apiVersion":"apps/v1","kind":"Deployment",`+ - `"metadata":{"name":%q,"namespace":%q,"uid":%q,"resourceVersion":%q}}`, name, namespace, uid, rv) - return auditv1.Event{ - AuditID: "audit-1", - Verb: verb, - Stage: auditv1.StageResponseComplete, - StageTimestamp: metav1.MicroTime{Time: time.Now()}, - User: authnv1.UserInfo{Username: username}, - ObjectRef: &auditv1.ObjectReference{ - APIGroup: "apps", - APIVersion: "v1", - Resource: "deployments", - Namespace: namespace, - Name: name, - UID: k8stypes.UID(uid), - }, - ResponseObject: &runtime.Unknown{Raw: []byte(body)}, - } -} - -func appsDeploymentGVR() schema.GroupVersionResource { - return schema.GroupVersionResource{Group: "apps", Version: "v1", Resource: "deployments"} -} - -func TestAttributionIndex_RecordAndLookupExact(t *testing.T) { - idx := newTestAttributionIndex(t) - ctx := context.Background() - - require.NoError(t, idx.RecordFact(ctx, "default", mutationEvent("update", "uid-1", "101", "alice"))) - - fact, ok := idx.LookupAuthor(ctx, "default", appsDeploymentGVR(), "uid-1", "101", true) - require.True(t, ok) - require.Equal(t, "alice", fact.Author) - require.Equal(t, "101", fact.ResourceVersion) - require.Equal(t, "apps/deployments", fact.GroupResource) - require.Equal(t, "team-a", fact.Namespace) - require.Equal(t, "web", fact.Name) - require.Equal(t, "uid-1", fact.UID) - require.False(t, fact.IsServiceAccount) -} - -func TestAttributionIndex_LookupByUIDWhenRVDiffers(t *testing.T) { - idx := newTestAttributionIndex(t) - ctx := context.Background() - - require.NoError(t, idx.RecordFact(ctx, "default", mutationEvent("delete", "uid-1", "101", "alice"))) - - // Watch DELETE lands at a later RV; the uid-latest :last pointer still resolves the - // author (a delete is not exact-capable, so it may consult :last). - fact, ok := idx.LookupAuthor(ctx, "default", appsDeploymentGVR(), "uid-1", "999", false) - require.True(t, ok) - require.Equal(t, "alice", fact.Author) -} - -func TestAttributionIndex_LookupResolutionWeakWhenExactMisses(t *testing.T) { - idx := newTestAttributionIndex(t) - ctx := context.Background() - - require.NoError(t, idx.RecordFact(ctx, "default", mutationEvent("delete", "uid-1", "101", "alice"))) - - resolution := idx.LookupAuthorResolution(ctx, "default", appsDeploymentGVR(), "uid-1", "999", false) - require.Equal(t, AttributionWeak, resolution.Result) - require.Equal(t, "alice", resolution.Fact.Author) -} - -func TestAttributionIndex_ExactCapableDoesNotFallThroughToLast(t *testing.T) { - idx := newTestAttributionIndex(t) - ctx := context.Background() - - // alice's write seeds both the exact key (uid-1:101) and the :last pointer. - require.NoError(t, idx.RecordFact(ctx, "default", mutationEvent("update", "uid-1", "101", "alice"))) - - // An exact-capable event at a different RV whose exact key is absent must NOT borrow - // the :last author — it is absent and ships as committer. - res := idx.LookupAuthorResolution(ctx, "default", appsDeploymentGVR(), "uid-1", "202", true) - require.Equal(t, AttributionAbsent, res.Result) - - // The same miss for a known RV-mismatch event DOES consult :last. - weak := idx.LookupAuthorResolution(ctx, "default", appsDeploymentGVR(), "uid-1", "202", false) - require.Equal(t, AttributionWeak, weak.Result) - require.Equal(t, "alice", weak.Fact.Author) -} - -func TestAttributionIndex_BurstKeepsEachWritePrecise(t *testing.T) { - idx := newTestAttributionIndex(t) - ctx := context.Background() - - // Two authors write the same object in a burst at distinct RVs. - require.NoError(t, idx.RecordFact(ctx, "default", mutationEvent("update", "uid-1", "1", "alice"))) - require.NoError(t, idx.RecordFact(ctx, "default", mutationEvent("update", "uid-1", "2", "bob"))) - - // Each watch event hits its own immutable exact key → both precise, no conflict. - f1, ok := idx.LookupAuthor(ctx, "default", appsDeploymentGVR(), "uid-1", "1", true) - require.True(t, ok) - require.Equal(t, "alice", f1.Author) - f2, ok := idx.LookupAuthor(ctx, "default", appsDeploymentGVR(), "uid-1", "2", true) - require.True(t, ok) - require.Equal(t, "bob", f2.Author) - - // :last is last-writer-wins (bob), consulted only by an RV-mismatch event. - fl, ok := idx.LookupAuthor(ctx, "default", appsDeploymentGVR(), "uid-1", "999", false) - require.True(t, ok) - require.Equal(t, "bob", fl.Author) -} - -func TestAttributionIndex_NoUIDFactWritesRVKeyOnly(t *testing.T) { - idx, mr := newTestAttributionIndexWithRedis(t) - ctx := context.Background() - - require.NoError(t, idx.RecordFact(ctx, "default", mutationEvent("update", "", "202", "alice"))) - - // The §5 escape hatch: a no-UID fact writes the type-scoped rv-only key and no - // object keys. - require.True(t, mr.Exists(idx.factKeyRV("default", "apps/deployments", "202"))) - require.False(t, mr.Exists(idx.factKeyLast("default", "apps/deployments", ""))) - - // An exact-capable watch event (which carries a UID) joins it via the rv-only fallback. - res := idx.LookupAuthorResolution(ctx, "default", appsDeploymentGVR(), "uid-live", "202", true) - require.Equal(t, AttributionWeak, res.Result) - require.Equal(t, "alice", res.Fact.Author) -} - -func TestAttributionIndex_UIDFactWritesNoDeadRVKey(t *testing.T) { - idx, mr := newTestAttributionIndexWithRedis(t) - ctx := context.Background() - - require.NoError(t, idx.RecordFact(ctx, "default", mutationEvent("update", "uid-1", "303", "alice"))) - - require.True(t, mr.Exists(idx.factKeyExact("default", "apps/deployments", "uid-1", "303"))) - require.True(t, mr.Exists(idx.factKeyLast("default", "apps/deployments", "uid-1"))) - require.False(t, mr.Exists(idx.factKeyRV("default", "apps/deployments", "303")), - "a UID-bearing fact's rv-only key would be dead, so it is not written") -} - -func TestAttributionIndex_ServiceAccountFlagged(t *testing.T) { - idx := newTestAttributionIndex(t) - ctx := context.Background() - - require.NoError(t, idx.RecordFact(ctx, "default", mutationEvent("update", "uid-1", "303", - "system:serviceaccount:flux-system:kustomize-controller"))) - - fact, ok := idx.LookupAuthor(ctx, "default", appsDeploymentGVR(), "uid-1", "303", true) - require.True(t, ok) - require.True(t, fact.IsServiceAccount) -} - -func TestAttributionIndex_NoUserIsNoOp(t *testing.T) { - idx := newTestAttributionIndex(t) - ctx := context.Background() - - require.NoError(t, idx.RecordFact(ctx, "default", mutationEvent("update", "uid-1", "101", ""))) - - _, ok := idx.LookupAuthor(ctx, "default", appsDeploymentGVR(), "uid-1", "101", true) - require.False(t, ok) -} - -func TestAttributionIndex_LookupMiss(t *testing.T) { - idx := newTestAttributionIndex(t) - _, ok := idx.LookupAuthor(context.Background(), "default", appsDeploymentGVR(), "uid-x", "1", true) - require.False(t, ok) -} - -func TestAttributionIndex_AgedOutFactIsAbsent(t *testing.T) { - store, mr := newTestRedisStoreWithRedis(t) - idx := store.AttributionIndex(time.Minute) - - ctx := context.Background() - require.NoError(t, idx.RecordFact(ctx, "default", mutationEvent("update", "uid-1", "101", "alice"))) - - mr.FastForward(time.Minute + time.Second) - - // No tombstone: an aged-out fact is indistinguishable from one that never arrived. - resolution := idx.LookupAuthorResolution(ctx, "default", appsDeploymentGVR(), "uid-1", "101", true) - require.Equal(t, AttributionAbsent, resolution.Result) -} - -func TestAttributionIndex_FactLifecycleMetrics(t *testing.T) { - reader, err := telemetry.InitTestExporter() - require.NoError(t, err) - idx := newTestAttributionIndex(t) - ctx := context.Background() - - require.NoError(t, idx.RecordFact(ctx, "default", mutationEvent("update", "uid-1", "101", "alice"))) - _ = idx.LookupAuthorResolution(ctx, "default", appsDeploymentGVR(), "uid-1", "101", true) - - written, ok := telemetry.CollectInt64Sum(reader, "gitopsreverser_attribution_fact_events_total", - map[string]string{"op": "written"}) - require.True(t, ok) - require.Equal(t, int64(1), written) - matched, ok := telemetry.CollectInt64Sum(reader, "gitopsreverser_attribution_fact_events_total", - map[string]string{"op": "matched"}) - require.True(t, ok) - require.Equal(t, int64(1), matched) - // One exact fact writes two keys: the immutable exact key and the :last pointer. - size, ok := telemetry.CollectInt64Sum(reader, "gitopsreverser_attribution_fact_index_size", nil) - require.True(t, ok) - require.Equal(t, int64(2), size) -} - -func TestAttributionIndex_RecordFactNoOpCases(t *testing.T) { - idx := newTestAttributionIndex(t) - ctx := context.Background() - - // No objectRef → nothing to key on. - require.NoError( - t, - idx.RecordFact(ctx, "default", auditv1.Event{Verb: "create", User: authnv1.UserInfo{Username: "a"}}), - ) - - // Empty resource → cannot build a key. - require.NoError(t, idx.RecordFact(ctx, "default", auditv1.Event{ - Verb: "create", - User: authnv1.UserInfo{Username: "a"}, - ObjectRef: &auditv1.ObjectReference{APIGroup: "apps", Name: "web"}, - })) - - // No resolvable name → no author can be attributed to an object. - require.NoError(t, idx.RecordFact(ctx, "default", auditv1.Event{ - Verb: "create", - User: authnv1.UserInfo{Username: "a"}, - ObjectRef: &auditv1.ObjectReference{APIGroup: "apps", Resource: "deployments"}, - })) -} - -func TestRedisStore_Ping(t *testing.T) { - store := newTestRedisStore(t) - require.NoError(t, store.Ping(context.Background())) -} - -func TestRedisStore_WatchCursorRoundTrip(t *testing.T) { - store, mr := newTestRedisStoreWithRedis(t) - ctx := context.Background() - gvr := appsDeploymentGVR() - - _, ok := store.LookupWatchCursor(ctx, "uid-1", gvr, "apps") - require.False(t, ok) - - require.NoError(t, store.RecordWatchCursor(ctx, "uid-1", gvr, "apps", "42")) - got, ok := store.LookupWatchCursor(ctx, "uid-1", gvr, "apps") - require.True(t, ok) - require.Equal(t, "42", got) - - // The cursor carries watchCursorTTL and is never deleted explicitly; it expires - // once a watch has been gone longer than the TTL. - require.Equal(t, watchCursorTTL, mr.TTL(store.watchCursorKey("uid-1", gvr, "apps"))) - mr.FastForward(watchCursorTTL + time.Second) - _, ok = store.LookupWatchCursor(ctx, "uid-1", gvr, "apps") - require.False(t, ok) -} - -func TestRedisStore_WatchCursorIsolatedByGitTargetUID(t *testing.T) { - store := newTestRedisStore(t) - ctx := context.Background() - gvr := appsDeploymentGVR() - - require.NoError(t, store.RecordWatchCursor(ctx, "uid-old", gvr, "apps", "42")) - - // A GitTarget recreated under the same namespace/name but a new UID must not - // inherit its predecessor's cursor. - _, ok := store.LookupWatchCursor(ctx, "uid-new", gvr, "apps") - require.False(t, ok) - - got, ok := store.LookupWatchCursor(ctx, "uid-old", gvr, "apps") - require.True(t, ok) - require.Equal(t, "42", got) -} - -func TestRedisStore_WatchCursorIgnoresEmptyResourceVersion(t *testing.T) { - store := newTestRedisStore(t) - ctx := context.Background() - - require.NoError(t, store.RecordWatchCursor(ctx, "uid-1", appsDeploymentGVR(), "apps", "")) - _, ok := store.LookupWatchCursor(ctx, "uid-1", appsDeploymentGVR(), "apps") - require.False(t, ok) -} - -func TestAttributionIndex_FactTTLConfigurable(t *testing.T) { - store, mr := newTestRedisStoreWithRedis(t) - idx := store.AttributionIndex(5 * time.Minute) - - require.NoError( - t, - idx.RecordFact(context.Background(), "default", mutationEvent("update", "uid-1", "101", "alice")), - ) - - for _, key := range []string{ - idx.factKeyExact("default", "apps/deployments", "uid-1", "101"), - idx.factKeyLast("default", "apps/deployments", "uid-1"), - } { - require.Equal(t, 5*time.Minute, mr.TTL(key), "fact key %q", key) - } -} - -func TestAttributionIndex_FactTTLDefaultsWhenUnset(t *testing.T) { - store, mr := newTestRedisStoreWithRedis(t) - idx := store.AttributionIndex(0) - - require.NoError( - t, - idx.RecordFact(context.Background(), "default", mutationEvent("update", "uid-1", "101", "alice")), - ) - - require.Equal(t, DefaultAttributionFactTTL, mr.TTL(idx.factKeyExact("default", "apps/deployments", "uid-1", "101"))) -} - -func TestEscapeKeyField(t *testing.T) { - cases := []struct{ in, want string }{ - {"web", "web"}, - {"101", "101"}, - {"rbac.authorization.k8s.io", "rbac.authorization.k8s.io"}, - {"system:node-proxier", "system%3Anode-proxier"}, - {"a%b", "a%25b"}, - {"%3A", "%253A"}, // a literal "%3A" must stay distinct from an escaped colon - {"", ""}, - } - for _, c := range cases { - require.Equal(t, c.want, escapeKeyField(c.in), "escapeKeyField(%q)", c.in) - } -} - -func TestGroupResourceKey(t *testing.T) { - require.Equal(t, "configmaps", groupResourceKey("", "configmaps")) - require.Equal(t, "apps/deployments", groupResourceKey("apps", "deployments")) - require.Equal(t, "rbac.authorization.k8s.io/roles", groupResourceKey("rbac.authorization.k8s.io", "roles")) -} - -// rawObject wraps a JSON body the way the audit pipeline delivers request/response bodies. -func rawObject(body string) *runtime.Unknown { - return &runtime.Unknown{Raw: []byte(body)} -} - -// deploymentBody renders a minimal Deployment whose metadata.resourceVersion is rv. -func deploymentBody(rv string) string { - return fmt.Sprintf(`{"apiVersion":"apps/v1","kind":"Deployment",`+ - `"metadata":{"name":"web","namespace":"team-a","uid":"uid-1","resourceVersion":%q}}`, rv) -} - -// TestResourceVersionFromEvent_Precedence pins the RV precedence the join depends on. The RV is -// half of the fact key, so reading the wrong one files the fact under an object version that will -// never be looked up — the write silently ships the committer instead of its real author. Only the -// POST-write RV identifies the version a mutation produced: that lives in responseObject. -// requestObject carries the PRE-write RV and must never be consulted, and objectRef.resourceVersion -// is usually the empty precondition RV on writes, so it is a last resort only. -func TestResourceVersionFromEvent_Precedence(t *testing.T) { - cases := []struct { - name string - mutate func(*auditv1.Event) - wantRV string - wantWhy string - }{ - { - name: "response object wins over a different objectRef RV", - mutate: func(e *auditv1.Event) { - e.ResponseObject = rawObject(deploymentBody("202")) - e.ObjectRef.ResourceVersion = "101" - }, - wantRV: "202", - wantWhy: "the post-write RV in the response body is authoritative", - }, - { - name: "request object is ignored even when the response object has none", - mutate: func(e *auditv1.Event) { - e.RequestObject = rawObject(deploymentBody("101")) - e.ResponseObject = nil - e.ObjectRef.ResourceVersion = "" - }, - wantRV: "", - wantWhy: "requestObject holds the pre-write RV and is never a source", - }, - { - name: "request object never outranks the response object", - mutate: func(e *auditv1.Event) { - e.RequestObject = rawObject(deploymentBody("101")) - e.ResponseObject = rawObject(deploymentBody("202")) - }, - wantRV: "202", - wantWhy: "responseObject is consulted first and short-circuits", - }, - { - name: "objectRef is the fallback when the response object is absent", - mutate: func(e *auditv1.Event) { - e.ResponseObject = nil - e.ObjectRef.ResourceVersion = "101" - }, - wantRV: "101", - wantWhy: "objectRef is the last resort, not the first choice", - }, - { - name: "objectRef is the fallback when the response body carries no RV", - mutate: func(e *auditv1.Event) { - e.ResponseObject = rawObject(`{"metadata":{"name":"web"}}`) - e.ObjectRef.ResourceVersion = "101" - }, - wantRV: "101", - wantWhy: "a shallow body yields nothing, so the fallback still applies", - }, - { - name: "objectRef is the fallback when the response body is malformed", - mutate: func(e *auditv1.Event) { - e.ResponseObject = rawObject(`{"metadata":`) - e.ObjectRef.ResourceVersion = "101" - }, - wantRV: "101", - wantWhy: "an unparseable body must not poison the fallback", - }, - { - name: "empty precondition RV on objectRef yields nothing", - mutate: func(e *auditv1.Event) { - e.ResponseObject = nil - e.ObjectRef.ResourceVersion = "" - }, - wantRV: "", - wantWhy: "writes usually leave objectRef.resourceVersion empty", - }, - { - name: "nil objectRef and nil response object yield nothing", - mutate: func(e *auditv1.Event) { - e.ResponseObject = nil - e.ObjectRef = nil - }, - wantRV: "", - wantWhy: "collection verbs and deletes legitimately have no RV", - }, - { - name: "nil objectRef does not fall through to the request object", - mutate: func(e *auditv1.Event) { - e.ResponseObject = nil - e.ObjectRef = nil - e.RequestObject = rawObject(deploymentBody("101")) - }, - wantRV: "", - wantWhy: "requestObject stays ignored on every path", - }, - } - - for _, c := range cases { - t.Run(c.name, func(t *testing.T) { - event := mutationEvent("update", "uid-1", "202", "alice") - c.mutate(&event) - require.Equal(t, c.wantRV, resourceVersionFromEvent(event), c.wantWhy) - }) - } -} - -// TestRVFromRawObject_Cases covers the body shapes the audit stream actually delivers. Every -// non-answer must be "" rather than a partial or panicking read: a truncated or bodyless audit -// event has to degrade into "no RV recorded", not into a bogus RV that keys a fact nobody finds. -func TestRVFromRawObject_Cases(t *testing.T) { - cases := []struct { - name string - obj *runtime.Unknown - want string - }{ - {name: "nil object", obj: nil, want: ""}, - {name: "nil raw bytes", obj: &runtime.Unknown{}, want: ""}, - {name: "zero-length raw bytes", obj: &runtime.Unknown{Raw: []byte{}}, want: ""}, - {name: "malformed json", obj: rawObject(`{"metadata":{"resourceVersion":`), want: ""}, - {name: "non-object json", obj: rawObject(`"a string"`), want: ""}, - {name: "empty json object", obj: rawObject(`{}`), want: ""}, - {name: "object without metadata", obj: rawObject(`{"kind":"Deployment"}`), want: ""}, - {name: "metadata without resourceVersion", obj: rawObject(`{"metadata":{"name":"web"}}`), want: ""}, - {name: "explicitly empty resourceVersion", obj: rawObject(`{"metadata":{"resourceVersion":""}}`), want: ""}, - {name: "well-formed body", obj: rawObject(deploymentBody("101")), want: "101"}, - { - name: "resourceVersion alongside unknown fields", - obj: rawObject(`{"spec":{"replicas":3},"metadata":{"name":"web","resourceVersion":"99"}}`), - want: "99", - }, - } - - for _, c := range cases { - t.Run(c.name, func(t *testing.T) { - require.Equal(t, c.want, rvFromRawObject(c.obj)) - }) - } -} - -// TestAttributionIndex_CrossClusterIsolation is the multi-cluster centerpiece: two clusters -// record the SAME object identity (uid, rv) with different authors, and each cluster's read joins -// ONLY its own fact — never the other's. A third cluster that recorded nothing misses (ships -// committer) instead of borrowing a neighbor's author. -func TestAttributionIndex_CrossClusterIsolation(t *testing.T) { - idx := newTestAttributionIndex(t) - ctx := context.Background() - - require.NoError(t, idx.RecordFact(ctx, "prod-eu-1", mutationEvent("update", "uid-1", "101", "alice"))) - require.NoError(t, idx.RecordFact(ctx, "prod-us-1", mutationEvent("update", "uid-1", "101", "bob"))) - - a, ok := idx.LookupAuthor(ctx, "prod-eu-1", appsDeploymentGVR(), "uid-1", "101", true) - require.True(t, ok) - require.Equal(t, "alice", a.Author) - - b, ok := idx.LookupAuthor(ctx, "prod-us-1", appsDeploymentGVR(), "uid-1", "101", true) - require.True(t, ok) - require.Equal(t, "bob", b.Author) - - _, ok = idx.LookupAuthor(ctx, "default", appsDeploymentGVR(), "uid-1", "101", true) - require.False(t, ok, "a cluster with no fact for this identity must miss, not cross-join") -} - -// TestAttributionIndex_RVOnlyHatchIsClusterScoped proves the correctness fix: the no-UID rv-only -// hatch is keyed by cluster, so the same RV in two clusters resolves to each cluster's own author -// (RV is not globally unique). -func TestAttributionIndex_RVOnlyHatchIsClusterScoped(t *testing.T) { - idx := newTestAttributionIndex(t) - ctx := context.Background() - - require.NoError(t, idx.RecordFact(ctx, "prod-eu-1", mutationEvent("update", "", "202", "alice"))) - require.NoError(t, idx.RecordFact(ctx, "prod-us-1", mutationEvent("update", "", "202", "bob"))) - - eu := idx.LookupAuthorResolution(ctx, "prod-eu-1", appsDeploymentGVR(), "uid-live", "202", true) - require.Equal(t, AttributionWeak, eu.Result) - require.Equal(t, "alice", eu.Fact.Author) - - us := idx.LookupAuthorResolution(ctx, "prod-us-1", appsDeploymentGVR(), "uid-live", "202", true) - require.Equal(t, "bob", us.Fact.Author, "same RV in another cluster must not leak across") -} - -// TestAttributionIndex_SingleProviderMatchesBareInstall proves a single-(default)-provider install -// round-trips correctly: what RecordFact writes under "default" is exactly what a "default" read -// joins, so a bare single-cluster install behaves as before the cluster dimension existed. -func TestAttributionIndex_SingleProviderMatchesBareInstall(t *testing.T) { - idx := newTestAttributionIndex(t) - ctx := context.Background() - require.NoError(t, idx.RecordFact(ctx, "default", mutationEvent("update", "uid-1", "101", "alice"))) - fact, ok := idx.LookupAuthor(ctx, "default", appsDeploymentGVR(), "uid-1", "101", true) - require.True(t, ok) - require.Equal(t, "alice", fact.Author) -} - -func TestAttributionIndex_FactKeyReadableFormat(t *testing.T) { - idx := newTestAttributionIndex(t) - - require.Equal(t, "gitops-reverser:author:v1:audit:route:default:apps/deployments:object:uid-1:101", - idx.factKeyExact("default", "apps/deployments", "uid-1", "101")) - require.Equal(t, "gitops-reverser:author:v1:audit:route:default:apps/deployments:object:uid-1:last", - idx.factKeyLast("default", "apps/deployments", "uid-1")) - require.Equal(t, "gitops-reverser:author:v1:audit:route:default:apps/deployments:rv:101", - idx.factKeyRV("default", "apps/deployments", "101")) - - // A remote provider keys under its own name, so its facts never collide with the local ones. - require.Equal(t, "gitops-reverser:author:v1:audit:route:prod-eu-1:apps/deployments:object:uid-1:101", - idx.factKeyExact("prod-eu-1", "apps/deployments", "uid-1", "101")) - - // The core group drops the group segment. - require.Equal(t, "gitops-reverser:author:v1:audit:route:default:configmaps:object:uid-2:last", - idx.factKeyLast("default", "configmaps", "uid-2")) -} - -func TestRedisStore_WatchCursorKeyReadableFormat(t *testing.T) { - store := newTestRedisStore(t) - gvr := schema.GroupVersionResource{Group: "apps", Version: "v1", Resource: "deployments"} - - require.Equal(t, "gitops-reverser:watch:v1:target:gtuid-3:apps/deployments:namespace:team-a:last-rv", - store.watchCursorKey("gtuid-3", gvr, "team-a")) - - // A cluster-wide watch (empty namespace) uses the cluster scope segment, and the GVR - // version is dropped. - require.Equal(t, "gitops-reverser:watch:v1:target:gtuid-3:configmaps:cluster:last-rv", - store.watchCursorKey("gtuid-3", coreConfigmapsGVR(), "")) -} - -func TestNewRedisStore_RequiresAddr(t *testing.T) { - _, err := NewRedisStore(RedisStoreConfig{}) - require.Error(t, err) -} - -// TestAttributionIndex_SharedAuditRouteJoinsAcrossProviders is the reported bug at the keyspace -// layer. An API server has one audit webhook backend and posts under ONE route, so a fact recorded -// on that route must be readable by every ClusterProvider that declares it, whatever those -// providers are named. Before the route existed, each provider read under its own name and only the -// routed one ever matched. -func TestAttributionIndex_SharedAuditRouteJoinsAcrossProviders(t *testing.T) { - idx := newTestAttributionIndex(t) - ctx := context.Background() - - // The local API server posts to /audit-webhook/default, so the fact lands on that route only. - require.NoError(t, idx.RecordFact(ctx, "default", mutationEvent("update", "uid-1", "101", "alice"))) - - // A dedicated in-cluster provider named srcns-delegating declares auditRoute: default, so its - // GitTargets read the same partition and resolve the same author. - fact, ok := idx.LookupAuthor(ctx, "default", appsDeploymentGVR(), "uid-1", "101", true) - require.True(t, ok) - require.Equal(t, "alice", fact.Author) - - // A provider that did NOT declare the route reads its own name and finds nothing. This is the - // exact failure the bug report measured, kept as a test so the fix cannot silently regress into - // "every route resolves everything". - _, ok = idx.LookupAuthor(ctx, "srcns-delegating", appsDeploymentGVR(), "uid-1", "101", true) - require.False(t, ok, "a route nobody wrote to must miss, or the partition means nothing") -} - -// TestAttributionIndex_WriteFactKeysByFactShape pins which keys each fact shape writes, which is -// where the v3 schema's §5 escape hatch lives. The rules are not symmetric and the asymmetry is -// deliberate: a UID-bearing fact's rv-only key would be DEAD, because the watch side always carries -// a UID and resolves via object::… first, so writing one would be a key nobody ever reads. -func TestAttributionIndex_WriteFactKeysByFactShape(t *testing.T) { - const route, gr = "prod-eu-1", "apps/deployments" - raw := []byte(`{"author":"alice"}`) - - tests := []struct { - name string - uid, rv string - wantWrote bool - }{ - { - name: "uid and rv write the immutable exact key and the last pointer", - uid: "uid-1", rv: "101", wantWrote: true, - }, - { - name: "uid without rv writes only the last pointer", - uid: "uid-1", wantWrote: true, - }, - { - name: "rv without uid writes only the rv-only hatch", - rv: "101", wantWrote: true, - }, - { - name: "neither uid nor rv writes nothing at all", - // An event that carries no joinable identity cannot ever be matched, so recording it - // would leave a key that only expires. Reporting wrote=false also keeps the "written" - // fact-event counter honest. - wantWrote: false, - }, - } - - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - idx, mr := newTestAttributionIndexWithRedis(t) - ctx := context.Background() - - wrote, err := idx.writeFactKeys(ctx, route, gr, tt.uid, tt.rv, raw) - require.NoError(t, err) - require.Equal(t, tt.wantWrote, wrote) - - exists := func(key string) bool { - _, getErr := mr.Get(key) - return getErr == nil - } - require.Equal(t, tt.uid != "" && tt.rv != "", exists(idx.factKeyExact(route, gr, tt.uid, tt.rv)), - "the exact key needs both a uid and the rv that write produced") - require.Equal(t, tt.uid != "", exists(idx.factKeyLast(route, gr, tt.uid)), - "the last-writer pointer is keyed by uid alone") - require.Equal(t, tt.uid == "" && tt.rv != "", exists(idx.factKeyRV(route, gr, tt.rv)), - "the rv-only hatch exists only for a fact that has no uid") - }) - } -} diff --git a/internal/queue/author_fact.go b/internal/queue/author_fact.go index 73c74d8c..5c045624 100644 --- a/internal/queue/author_fact.go +++ b/internal/queue/author_fact.go @@ -4,19 +4,32 @@ package queue import ( "context" + "encoding/json" + "errors" + "fmt" "net/url" "strings" "time" "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/metric" + "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/apimachinery/pkg/types" auditv1 "k8s.io/apiserver/pkg/apis/audit/v1" "github.com/ConfigButler/gitops-reverser/internal/auditutil" "github.com/ConfigButler/gitops-reverser/internal/telemetry" ) +// DefaultAttributionFactTTL is how long an attribution fact stays joinable while it waits for the +// matching watch event. It bounds the stream's retention horizon and the in-memory index together, +// and it doubles as the follower's replay horizon, so a restart warms the index with exactly the +// window that is still usable. After it elapses a miss is simply "absent": there is no tombstone, +// so an aged-out fact is indistinguishable from one that never arrived. +// Configurable via --author-attribution-ttl. +const DefaultAttributionFactTTL = 10 * time.Minute + // DefaultCollectionUIDCap is how many uids a collection fact may carry before the set is dropped // and the join falls back to scope matching. // @@ -31,6 +44,190 @@ const DefaultCollectionUIDCap = 10000 // labelSelectorQueryParam is where a collection request states which objects it meant. const labelSelectorQueryParam = "labelSelector" +// serviceAccountUserPrefix is how the API server spells a service account in an audit event's user. +const serviceAccountUserPrefix = "system:serviceaccount:" + +// AttributionResult is the bounded resolver outcome recorded for each watch event. The set is the +// join's tier table, so a reading of attribution_resolutions_total{tier} says which evidence named +// the author, not merely that one was named. It is the source of truth for that label: a lookup +// that could not say which tier answered is the thing the tier label exists to fix, so the split +// lives in the enum rather than at the metric boundary. +// +// WHO was named is a separate question and a separate label — see ActorKind. The two used to be +// crammed into one value (exact_user against exact_serviceaccount), which made counting exact +// resolutions a sum of two series and made the actor kind unaskable of every other tier. +type AttributionResult string + +const ( + // AttributionExact is an exact UID+resourceVersion match: this actor produced this exact version. + AttributionExact AttributionResult = "exact" + // AttributionLatest is the uid-latest tier — the object's own last write or its own delete fact, + // keyed by uid alone. It is the tier the removal path turns on: a match here that describes a + // WRITE is held as a fallback while the wait continues for evidence about the deletion. + AttributionLatest AttributionResult = "latest" + // AttributionResourceVersion is the rv-only escape hatch: a fact that carried a resourceVersion + // and no uid, matched on that version alone. It and AttributionLatest were one value ("weak") + // and are different evidence, which is why they are now two. + AttributionResourceVersion AttributionResult = "resource_version" + // AttributionCollectionUID is a removal matched to a deletecollection fact whose uid set + // contains this object. There is no over-attribution risk in it: either the API server said it + // deleted this object, or it did not. + AttributionCollectionUID AttributionResult = "collection_uid" + // AttributionName is a match on (namespace, name) for a fact that carries neither a uid nor a + // resourceVersion. It is the tier of last resort for a type whose audit event cannot express + // object identity: the kube-apiserver proxies an aggregated-API request and never decodes the + // response, so the objectRef carries the name from the URL path and nothing else, and there is no + // body to backfill from. Measured in corpus flunder/aggregated-api-delete. + // + // It ranks below every other per-object tier because a name is REUSED after a delete and + // recreate where a uid is not, so it can name the author of a previous object that held this + // name. The TTL is what bounds that: the wrong answer requires the recreate to happen inside it. + AttributionName AttributionResult = "name" + // AttributionCollectionScope is a removal matched to a deletecollection fact by scope alone — + // same type and namespace, selector accepting the object's labels, within the collection window. + // It is the weakest evidence the join has, which is why it is reached only when every more + // specific tier missed. It is also the tier that resolves what the deleted expander gave up on: + // a collection delete the API server sent no response body for. + AttributionCollectionScope AttributionResult = "collection_scope" + // AttributionAbsent means no usable author fact matched before the grace elapsed. + AttributionAbsent AttributionResult = "absent" +) + +// AuthorFact is the minimal attribution fact published per accepted, mutating audit event and read +// back by the watch-event resolver. It names an author candidate and carries the evidence the join +// needs to decide confidence; it is never object state. +// +// Every field here is either read by the join or printed when a fact is investigated. That is a +// deliberate bar, because a fact is not stored once: it is broadcast to every process following its +// type, held for the whole TTL, and replayed into memory on every restart, so a field nothing reads +// is paid for on all three. Three fields were removed for failing it: +// +// - the group/resource, which is the STREAM'S OWN NAME — the index takes the scope from the +// entry's key, never from the fact, so carrying it duplicated the routing on every entry; +// - the subresource, which no tier joins on and nothing logs. +// +// A stored is-service-account bool went the same way, for a different reason: it is not evidence, +// it is a prefix check on Author that the reader can do for itself (see ActorKind). +// +// Name was removed with the subresource, on the same observation — no tier read it — and is back, +// because that observation was true of the code and false of the domain. An aggregated-API write is +// audited with no uid and no resourceVersion, and the name from the URL path is the ONLY identity it +// carries, so a fact without it could not be joined at all for that whole population. "No code reads +// it" and "nothing could ever read it" are different claims, and only the second justifies dropping a +// field. +type AuthorFact struct { //nolint:recvcheck // UnmarshalJSON must take a pointer; every other method only reads. + Namespace string `json:"namespace,omitempty"` + UID string `json:"uid,omitempty"` + // Name is the object's name, and it feeds one tier only: the (namespace, name) join a fact with + // no uid and no resourceVersion is otherwise unreachable through. A collection fact clears it, + // because a collection request names no object. + Name string `json:"name,omitempty"` + // Author is the actor's username, and it is the ONE required field on the wire: a fact exists to + // name somebody, so a fact that names nobody is not a weak fact, it is not a fact. It is never + // empty and never null — see UnmarshalJSON, which refuses an entry carrying one. + Author string `json:"author"` + // DisplayName and Email are the actor's, when the API server supplied them. They are the only + // fields here that are not identity or evidence: they exist because a commit author is a name + // and an email, and re-deriving them at commit time would need a second lookup. + DisplayName string `json:"displayName,omitempty"` + Email string `json:"email,omitempty"` + Verb string `json:"verb,omitempty"` + // AuditID is the one field the join never reads and is kept anyway. It is what ties a commit + // authored by the wrong person back to the audit event that named them, which is the single + // question a mis-attribution investigation asks and the one thing nothing else in the system + // can answer. + AuditID string `json:"auditID,omitempty"` + ResourceVersion string `json:"resourceVersion,omitempty"` + StageTimestamp string `json:"stageTimestamp,omitempty"` + + // LabelSelector is the selector the request URI expressed, carried on a COLLECTION fact only. + // It is the intent the actor stated, and evaluating it against the object a watch event carries + // is a better test of membership than reading back a list the API server may not have sent. + // Empty means the collection covered everything of its type in its namespace, which is what + // --all means. + LabelSelector string `json:"labelSelector,omitempty"` + // UIDs is the set of objects a collection delete covered, reduced from the response body at the + // receiver, on a COLLECTION fact only. It is absent when the API server sent no body — a + // truncated, aggregated, or metadata-only response — and when the set was larger than the cap, + // in which case the join falls back to scope matching, which is already correct. + UIDs []string `json:"uids,omitempty"` +} + +// errFactWithoutAuthor is what an entry violating the fact contract decodes to. +var errFactWithoutAuthor = errors.New("attribution fact carries no author") + +// UnmarshalJSON decodes a fact and refuses one that names nobody, which is the whole wire contract: +// `author` must be present, a string, and non-empty. Missing, `null`, and `""` are the same +// violation and are all refused. +// +// Go cannot express "a string of at least one character" as a type — every type has a zero value +// that is constructible without going through any constructor, and `encoding/json` writes exported +// fields straight past one anyway — so the constraint lives at the only boundary that can hold it: +// the point where a fact written by somebody else enters this process. +// +// The refusal is deliberately at ENTRY granularity, not per fact. This operator's publish gate +// cannot produce an authorless fact (AuthorFactFromEvent refuses an event whose user is +// unresolvable, and counts it as no_attribution_fact), so an entry carrying one was written by +// something else: a different version, a different producer, or a hand-written entry. That is a +// protocol violation rather than a low-quality fact, and it is better counted and logged loudly — +// it lands on attribution_fact_stream_decode_errors_total with the stream and entry id — than +// half-absorbed by silently dropping one fact out of a batch. +func (f *AuthorFact) UnmarshalJSON(raw []byte) error { + // wire has AuthorFact's fields and tags but none of its methods, so decoding it does not recurse. + type wire AuthorFact + var decoded wire + if err := json.Unmarshal(raw, &decoded); err != nil { + return err + } + if decoded.Author == "" { + return fmt.Errorf("%w (auditID %q, verb %q)", errFactWithoutAuthor, decoded.AuditID, decoded.Verb) + } + *f = AuthorFact(decoded) + return nil +} + +// AuthorResolution is the structured result of an attribution lookup. +type AuthorResolution struct { + Fact AuthorFact + Result AttributionResult +} + +// ActorKind is the bounded kind of actor a resolution named. It is the same vocabulary +// commits_total{author_kind} uses, so the two metrics stop disagreeing about the shape of one +// distinction, and it is orthogonal to the tier: every tier can name either kind of actor, or none. +type ActorKind string + +const ( + // ActorKindUser is a human (or any non-service-account subject) named by the matched fact. + ActorKindUser ActorKind = "user" + // ActorKindServiceAccount is a named service account. + ActorKindServiceAccount ActorKind = "serviceaccount" + // ActorKindNone is no actor at all: nothing matched, or the fact that matched carried no author. + ActorKindNone ActorKind = "none" +) + +// ActorKind classifies the actor a fact names. It is derived rather than carried: the API server +// spells a service account one way and only one way, so a stored kind would be the same check, +// denormalized onto every fact and able to disagree with the name beside it. +func (f AuthorFact) ActorKind() ActorKind { + switch { + case f.Author == "": + return ActorKindNone + case strings.HasPrefix(f.Author, serviceAccountUserPrefix): + return ActorKindServiceAccount + default: + return ActorKindUser + } +} + +// ActorKind classifies the actor this resolution named, which is none when no fact matched. +func (r AuthorResolution) ActorKind() ActorKind { + if r.Result == AttributionAbsent { + return ActorKindNone + } + return r.Fact.ActorKind() +} + // AuthorFactFromEvent reduces one accepted, mutating audit event to the fact the stream carries, // reporting false when the event can never name an author. Only facts that WOULD have been stored // may be published: an event with no objectRef or no user produces nothing, or waiters are woken by @@ -43,7 +240,11 @@ const labelSelectorQueryParam = "labelSelector" // // The caller has already applied the intrinsic accept gate: reads, failures, dry runs, and // non-ResponseComplete stages never reach here. -func AuthorFactFromEvent(ctx context.Context, event auditv1.Event) (AuthorFact, schema.GroupResource, bool) { +func AuthorFactFromEvent( + ctx context.Context, + event auditv1.Event, + uidCap int, +) (AuthorFact, schema.GroupResource, bool) { if event.ObjectRef == nil || event.ObjectRef.Resource == "" { return AuthorFact{}, schema.GroupResource{}, false } @@ -61,24 +262,21 @@ func AuthorFactFromEvent(ctx context.Context, event auditv1.Event) (AuthorFact, groupResource := schema.GroupResource{Group: event.ObjectRef.APIGroup, Resource: event.ObjectRef.Resource} fact := AuthorFact{ - GroupResource: groupResourceKey(groupResource.Group, groupResource.Resource), - Namespace: identity.Namespace, - Name: identity.Name, - UID: string(identity.UID), - Author: user.Username, - DisplayName: user.DisplayName, - Email: user.Email, - Verb: event.Verb, - Subresource: event.ObjectRef.Subresource, - AuditID: string(event.AuditID), - ResourceVersion: resourceVersionFromEvent(event), - IsServiceAccount: strings.HasPrefix(user.Username, serviceAccountUserPrefix), + Namespace: identity.Namespace, + UID: string(identity.UID), + Name: identity.Name, + Author: user.Username, + DisplayName: user.DisplayName, + Email: user.Email, + Verb: event.Verb, + AuditID: string(event.AuditID), + ResourceVersion: resourceVersionFromEvent(event), } if !event.StageTimestamp.IsZero() { fact.StageTimestamp = event.StageTimestamp.UTC().Format(time.RFC3339Nano) } if collection { - describeCollection(ctx, &fact, event) + describeCollection(ctx, &fact, event, uidCap) } return fact, groupResource, true } @@ -91,12 +289,12 @@ func AuthorFactFromEvent(ctx context.Context, event auditv1.Event) (AuthorFact, // asymmetry the expander used to fight: audit reports the ONE request that was made, the watch // reports each of the N objects that changed, and the join belongs at the point where both are in // hand rather than in a receiver rebuilding N from one. -func describeCollection(ctx context.Context, fact *AuthorFact, event auditv1.Event) { - fact.Name = "" +func describeCollection(ctx context.Context, fact *AuthorFact, event auditv1.Event, uidCap int) { fact.UID = "" + fact.Name = "" fact.ResourceVersion = "" fact.LabelSelector = labelSelectorFromRequestURI(event.RequestURI) - fact.UIDs = collectionUIDs(ctx, event) + fact.UIDs = collectionUIDs(ctx, event, uidCap) } // labelSelectorFromRequestURI reads the selector a collection request expressed. It is better @@ -120,13 +318,16 @@ func labelSelectorFromRequestURI(requestURI string) string { // absent, hollow, or larger than the cap — all of which degrade the join to scope matching, which // is the floor that must work on its own. Crossing the cap is COUNTED, so "we fell back to scope" // is visible rather than inferred. -func collectionUIDs(ctx context.Context, event auditv1.Event) []string { +func collectionUIDs(ctx context.Context, event auditv1.Event, uidCap int) []string { + if uidCap <= 0 { + uidCap = DefaultCollectionUIDCap + } items := deleteCollectionItems(event.ResponseObject) if len(items) == 0 { return nil } - if len(items) > DefaultCollectionUIDCap { - recordCollectionDegraded(ctx, "uid_cap") + if len(items) > uidCap { + recordCollectionWithoutUIDSet(ctx, "uid_cap") return nil } uids := make([]string, 0, len(items)) @@ -136,18 +337,91 @@ func collectionUIDs(ctx context.Context, event auditv1.Event) []string { } } if len(uids) == 0 { - recordCollectionDegraded(ctx, "no_uids") + recordCollectionWithoutUIDSet(ctx, "no_uids") return nil } return uids } -// recordCollectionDegraded counts one collection fact that lost its uid set, under the bounded +// deleteCollectionItem is the per-object identity read from a deletecollection response list. It is +// all that survives of the deleted expander, and it survives for one job: reducing the body to the +// uid SET the fact carries. Nothing rebuilds N per-object facts from one request any more. +type deleteCollectionItem struct { + UID types.UID +} + +// deleteCollectionItems parses the per-object identities from a deletecollection response body. It +// accepts any list-shaped body (a typed "…List", a v1.List, or anything carrying an "items" array) +// and returns nil for a Status, hollow, or unparseable body — the join then falls back to scope +// matching, which is the floor that must work on its own. +func deleteCollectionItems(obj *runtime.Unknown) []deleteCollectionItem { + if obj == nil || len(obj.Raw) == 0 { + return nil + } + var envelope struct { + Items []struct { + Metadata struct { + UID types.UID `json:"uid"` + } `json:"metadata"` + } `json:"items"` + } + if err := json.Unmarshal(obj.Raw, &envelope); err != nil { + return nil + } + items := make([]deleteCollectionItem, 0, len(envelope.Items)) + for _, it := range envelope.Items { + items = append(items, deleteCollectionItem{UID: it.Metadata.UID}) + } + return items +} + +// resourceVersionFromEvent returns the event's ResourceVersion when one is available, or "" when it +// is not (deletes, collection verbs, shallow bodies). The post-write RV lives in the response +// object's metadata.resourceVersion; requestObject.resourceVersion is the pre-write RV on +// update-style requests, so it is intentionally ignored. objectRef.resourceVersion is usually the +// empty precondition RV on writes, so it is only the last resort. +func resourceVersionFromEvent(event auditv1.Event) string { + if rv := rvFromRawObject(event.ResponseObject); rv != "" { + return rv + } + if event.ObjectRef != nil { + return event.ObjectRef.ResourceVersion + } + return "" +} + +func rvFromRawObject(obj *runtime.Unknown) string { + if obj == nil || len(obj.Raw) == 0 { + return "" + } + var probe struct { + Metadata struct { + ResourceVersion string `json:"resourceVersion"` + } `json:"metadata"` + } + if err := json.Unmarshal(obj.Raw, &probe); err != nil { + return "" + } + return probe.Metadata.ResourceVersion +} + +// RecordFactsWritten counts facts appended to the fact log. It is called by the publish side, once +// per append rather than once per entry, so the counter measures facts rather than audit batches +// and stays comparable with the matched op on the other side of the join. +func RecordFactsWritten(ctx context.Context, count int) { + if telemetry.AttributionFactsTotal == nil || count <= 0 { + return + } + telemetry.AttributionFactsTotal.Add(ctx, int64(count), + metric.WithAttributes(attribute.String("op", factOpWritten))) +} + +// recordCollectionWithoutUIDSet counts one collection fact that lost its uid set, under the bounded // reason it lost it. -func recordCollectionDegraded(ctx context.Context, reason string) { - if telemetry.AttributionCollectionDegradedTotal == nil { +func recordCollectionWithoutUIDSet(ctx context.Context, reason string) { + if telemetry.AttributionCollectionWithoutUIDSetTotal == nil { return } - telemetry.AttributionCollectionDegradedTotal.Add(ctx, 1, + telemetry.AttributionCollectionWithoutUIDSetTotal.Add(ctx, 1, metric.WithAttributes(attribute.String("reason", reason))) } diff --git a/internal/queue/author_fact_test.go b/internal/queue/author_fact_test.go index 18663d94..cf4f7804 100644 --- a/internal/queue/author_fact_test.go +++ b/internal/queue/author_fact_test.go @@ -3,18 +3,51 @@ package queue import ( + "encoding/json" "fmt" "strings" "testing" + "time" "github.com/stretchr/testify/require" authnv1 "k8s.io/api/authentication/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + k8stypes "k8s.io/apimachinery/pkg/types" auditv1 "k8s.io/apiserver/pkg/apis/audit/v1" "github.com/ConfigButler/gitops-reverser/internal/telemetry" ) +// mutationEvent builds an apps/deployments event for team-a/web authored by username, +// whose objectRef + responseObject carry uid and resourceVersion rv. +func mutationEvent(verb, uid, rv, username string) auditv1.Event { + const namespace, name = "team-a", "web" + body := fmt.Sprintf(`{"apiVersion":"apps/v1","kind":"Deployment",`+ + `"metadata":{"name":%q,"namespace":%q,"uid":%q,"resourceVersion":%q}}`, name, namespace, uid, rv) + return auditv1.Event{ + AuditID: "audit-1", + Verb: verb, + Stage: auditv1.StageResponseComplete, + StageTimestamp: metav1.MicroTime{Time: time.Now()}, + User: authnv1.UserInfo{Username: username}, + ObjectRef: &auditv1.ObjectReference{ + APIGroup: "apps", + APIVersion: "v1", + Resource: "deployments", + Namespace: namespace, + Name: name, + UID: k8stypes.UID(uid), + }, + ResponseObject: &runtime.Unknown{Raw: []byte(body)}, + } +} + +func appsDeploymentGVR() schema.GroupVersionResource { + return schema.GroupVersionResource{Group: "apps", Version: "v1", Resource: "deployments"} +} + // collectionDeleteEvent is one name-less collection delete over count objects. func collectionDeleteEvent(selector string, count int) auditv1.Event { items := make([]string, 0, count) @@ -41,28 +74,28 @@ func collectionDeleteEvent(selector string, count int) auditv1.Event { } func TestAuthorFactFromEvent_CollectionCarriesScopeSelectorAndUIDs(t *testing.T) { - fact, groupResource, ok := AuthorFactFromEvent(t.Context(), collectionDeleteEvent("app%3Dweb", 2)) + fact, groupResource, ok := AuthorFactFromEvent(t.Context(), collectionDeleteEvent("app%3Dweb", 2), 0) require.True(t, ok, "a name-less deletecollection is the case that DOES produce a fact") require.Equal(t, "configmaps", groupResource.Resource) require.Equal(t, "team-a", fact.Namespace) require.Equal(t, "app=web", fact.LabelSelector) require.Equal(t, []string{"uid-0", "uid-1"}, fact.UIDs) - require.Empty(t, fact.Name) - require.Empty(t, fact.UID) + require.Empty(t, fact.UID, "a collection request names no object") } func TestAuthorFactFromEvent_UIDSetIsDroppedPastTheCapAndCounted(t *testing.T) { reader, err := telemetry.InitTestExporter() require.NoError(t, err) - fact, _, ok := AuthorFactFromEvent(t.Context(), collectionDeleteEvent("", DefaultCollectionUIDCap+1)) + fact, _, ok := AuthorFactFromEvent(t.Context(), collectionDeleteEvent("", DefaultCollectionUIDCap+1), 0) require.True(t, ok) // The fact degrades to scope matching, which is already correct — and says so in the metrics, // so "we fell back to scope" is visible rather than inferred. require.Nil(t, fact.UIDs) require.Empty(t, fact.LabelSelector) - degraded, found := telemetry.CollectInt64Sum(reader, "gitopsreverser_attribution_collection_degraded_total", + degraded, found := telemetry.CollectInt64Sum(reader, + "gitopsreverser_attribution_collection_without_uidset_total", map[string]string{"reason": "uid_cap"}) require.True(t, found) require.Equal(t, int64(1), degraded) @@ -74,7 +107,7 @@ func TestAuthorFactFromEvent_BodylessCollectionStillProducesAFact(t *testing.T) // The shape a production cluster with --audit-webhook-truncate-enabled actually sends, and the // one the old expander gave up on entirely. - fact, _, ok := AuthorFactFromEvent(t.Context(), event) + fact, _, ok := AuthorFactFromEvent(t.Context(), event, 0) require.True(t, ok) require.Nil(t, fact.UIDs) require.Equal(t, "alice", fact.Author) @@ -98,18 +131,17 @@ func TestAuthorFactFromEvent_EventsThatCanNameNobody(t *testing.T) { } for name, event := range cases { t.Run(name, func(t *testing.T) { - _, _, ok := AuthorFactFromEvent(t.Context(), event) + _, _, ok := AuthorFactFromEvent(t.Context(), event, 0) require.False(t, ok) }) } } func TestAuthorFactFromEvent_ObjectWriteCarriesTheIdentityTheJoinNeeds(t *testing.T) { - fact, groupResource, ok := AuthorFactFromEvent(t.Context(), mutationEvent("update", "uid-1", "101", "alice")) + fact, groupResource, ok := AuthorFactFromEvent(t.Context(), mutationEvent("update", "uid-1", "101", "alice"), 0) require.True(t, ok) require.Equal(t, "apps", groupResource.Group) require.Equal(t, "deployments", groupResource.Resource) - require.Equal(t, "apps/deployments", fact.GroupResource) require.Equal(t, "uid-1", fact.UID) require.Equal(t, "101", fact.ResourceVersion) require.Equal(t, "alice", fact.Author) @@ -118,3 +150,35 @@ func TestAuthorFactFromEvent_ObjectWriteCarriesTheIdentityTheJoinNeeds(t *testin require.Empty(t, fact.LabelSelector) require.Nil(t, fact.UIDs) } + +// TestAuthorFact_UnmarshalRefusesAFactThatNamesNobody pins the wire contract. A fact exists to name +// somebody, so `author` is the one required field, and missing, null, and empty are one violation +// rather than three. Go cannot state that as a type — every type has a constructible zero value, and +// encoding/json writes exported fields past any constructor — so the boundary holds it instead. +func TestAuthorFact_UnmarshalRefusesAFactThatNamesNobody(t *testing.T) { + refused := map[string]string{ + "author missing": `{"uid":"uid-1","verb":"update"}`, + "author null": `{"uid":"uid-1","author":null,"verb":"update"}`, + "author empty": `{"uid":"uid-1","author":"","verb":"update"}`, + } + for name, payload := range refused { + t.Run(name, func(t *testing.T) { + var fact AuthorFact + err := json.Unmarshal([]byte(payload), &fact) + require.ErrorIs(t, err, errFactWithoutAuthor) + }) + } + + // A named actor decodes intact, fields and all — the check refuses a fact, it does not filter one. + var fact AuthorFact + require.NoError(t, json.Unmarshal( + []byte(`{"uid":"uid-1","author":"alice","email":"a@x.io","verb":"update","uids":["a","b"]}`), &fact)) + require.Equal(t, "alice", fact.Author) + require.Equal(t, "a@x.io", fact.Email) + require.Equal(t, []string{"a", "b"}, fact.UIDs) + + // A batch is refused as a whole when any fact in it violates the contract: the entry is the unit + // the transport can skip, and a partially-absorbed batch would hide the violation. + _, err := decodeFactBatch([]byte(`[{"author":"alice"},{"author":""}]`)) + require.ErrorIs(t, err, errFactWithoutAuthor) +} diff --git a/internal/queue/fact_index.go b/internal/queue/fact_index.go index 90fdc623..3cdc6493 100644 --- a/internal/queue/fact_index.go +++ b/internal/queue/fact_index.go @@ -4,6 +4,7 @@ package queue import ( "context" + "math" "strings" "sync" "time" @@ -64,6 +65,14 @@ const ( evictionReasonTotal = "total" ) +// factOpWritten and factOpMatched are the bounded ops on the fact lifecycle counter: one fact +// appended to the log, and one joined by a watch event. Read together they say how much of what is +// published is ever used — the ratio that decides whether a type is worth following at all. +const ( + factOpWritten = "written" + factOpMatched = "matched" +) + // FactQuery is one watch event's identity, as the join reads it. It is everything the index needs // to try all five tiers, so a caller assembles it once rather than threading five arguments. type FactQuery struct { @@ -77,6 +86,9 @@ type FactQuery struct { // deletecollection whose scope covered it. Namespace string Labels map[string]string + // Name serves the name tier only, the floor reached when a fact carries neither a uid nor a + // resourceVersion. The watch side always knows it; only the audit side can be missing it. + Name string // ExactCapable is true for ADDED and MODIFIED, whose resourceVersion is the one the write // produced. A removal's is not, so it consults the weaker tiers the exact-capable events skip. ExactCapable bool @@ -158,12 +170,40 @@ func NewFactIndex(cfg FactIndexConfig) *FactIndex { // Apply stores one delivered entry's facts and wakes whoever was waiting for them. Facts are // applied in the order they were delivered, which is what makes the latest tier last-writer-wins // mean the last fact APPENDED rather than whichever goroutine reached the map first. +// A fact ages from when it was APPENDED, not from when this process happened to read it. The two +// differ by more than a hair in the case that matters most: the follower replays the whole retention +// window on start, so stamping those entries with the read time would hand every one of them a +// second full TTL and let a restart resurrect facts the horizon had already retired. A follower that +// falls behind, or a transport that hands back an entry its own retention should have dropped, lands +// in the same place. Reading the append time off the entry's position makes the TTL mean the same +// thing on both transports and on every delivery path, which is what SweepInterval bounding memory +// rather than correctness depends on. func (i *FactIndex) Apply(ctx context.Context, entry FactEntry) { scope := factScope{route: entry.Key.AuditRoute, groupResource: entry.Key.groupResource()} - now := time.Now() + at := entryAppendTime(entry.ID, time.Now()) for _, fact := range entry.Facts { - i.waiters.wake(i.store(ctx, scope, fact, now)) + i.waiters.wake(i.store(ctx, scope, fact, at)) + } +} + +// entryAppendTime reads the append time out of a transport position. Stream IDs are millisecond +// timestamps in both implementations, so a position IS a time and needs no side channel. +// +// It falls back to now for a position it cannot read, and clamps a future one: a fact must never +// age SLOWER than the clock because a transport handed back a malformed or skewed ID, which is the +// one direction that would extend a fact's life beyond its TTL rather than shorten it. +func entryAppendTime(id string, now time.Time) time.Time { + millis, _ := parseStreamID(id) + // A position past the int64 millisecond range is not a time this code can read, and neither is + // zero. Both fall back to now, which ages the fact from this moment rather than from never. + if millis == 0 || millis > math.MaxInt64 { + return now + } + at := time.UnixMilli(int64(millis)) + if at.After(now) { + return now } + return at } // Await resolves a watch event, waiting up to grace for a fact that has not been delivered yet. It @@ -174,15 +214,29 @@ func (i *FactIndex) Apply(ctx context.Context, entry FactEntry) { // BEFORE the index is read, so a fact applied in the gap between the two signals a waiter that is // already listening. Checking first and registering after loses exactly that fact — the race the // poll loop used to paper over by looking again. +// +// A match does not always end the wait. For a REMOVAL, the strongest fact present early is often +// the object's last WRITE, which says who edited it and nothing about who deleted it — and the +// watch event reliably beats the audit batch that carries the delete, which is the entire reason +// the grace window exists. Returning on that first match answered "who deleted this" with "who last +// edited it", every time an object was touched by someone else before being removed. Such a match +// is held as a FALLBACK instead: the wait continues for evidence about the deletion itself, and the +// fallback is returned only when the grace expires without any arriving. Attribution is never lost +// by waiting — the worst case returns exactly what returning early would have. func (i *FactIndex) Await(ctx context.Context, query FactQuery, grace time.Duration) AuthorResolution { waiter := i.waiters.register(query.waiterKeys()) defer i.waiters.unregister(waiter) + fallback := AuthorResolution{Result: AttributionAbsent} if resolution := i.Lookup(query); resolution.Result != AttributionAbsent { - return resolution + if !query.awaitsBetterEvidence(resolution) { + recordFactEvent(ctx, factOpMatched) + return resolution + } + fallback = resolution } if grace <= 0 { - return AuthorResolution{Result: AttributionAbsent} + return i.settle(ctx, fallback) } timer := time.NewTimer(grace) @@ -190,29 +244,88 @@ func (i *FactIndex) Await(ctx context.Context, query FactQuery, grace time.Durat for { select { case <-ctx.Done(): - return AuthorResolution{Result: AttributionAbsent} + return i.settle(ctx, fallback) case <-timer.C: - return AuthorResolution{Result: AttributionAbsent} + return i.settle(ctx, fallback) case <-waiter.ch: - if resolution := i.Lookup(query); resolution.Result != AttributionAbsent { + resolution := i.Lookup(query) + if resolution.Result == AttributionAbsent { + continue + } + if !query.awaitsBetterEvidence(resolution) { + recordFactEvent(ctx, factOpMatched) return resolution } + fallback = resolution } } } +// settle returns the fallback a removal was holding, counting it as the match it is. Waiting is +// never allowed to turn an attribution into an absence: a removal whose only evidence is the last +// write still names that writer once nothing better has arrived. +func (i *FactIndex) settle(ctx context.Context, fallback AuthorResolution) AuthorResolution { + if fallback.Result != AttributionAbsent { + recordFactEvent(ctx, factOpMatched) + } + return fallback +} + +// awaitsBetterEvidence reports whether a match should be held as a fallback rather than returned. +// +// It is true for exactly one shape: a REMOVAL matched to a fact that is not about a removal. The +// per-object tiers are last-writer-wins, so for a collection member — whose delete files one fact +// about the collection rather than one per object — they hold whoever edited it last. Both +// collection tiers are about the deletion itself and end the wait, as does a per-object fact whose +// own verb is a delete: that is the object's own removal fact, which is the strongest thing a +// removal can hope for. +func (q FactQuery) awaitsBetterEvidence(resolution AuthorResolution) bool { + if q.ExactCapable || resolution.Result == AttributionAbsent { + return false + } + switch resolution.Result { + case AttributionCollectionUID, AttributionCollectionScope: + return false + case AttributionExact, AttributionLatest, AttributionResourceVersion, + AttributionName, AttributionAbsent: + } + return !isRemovalVerb(resolution.Fact.Verb) +} + +// isRemovalVerb reports whether a fact describes a deletion rather than a write. +func isRemovalVerb(verb string) bool { + return strings.EqualFold(verb, "delete") || strings.EqualFold(verb, deleteCollectionVerb) +} + // Lookup reads the index once, trying the tiers strongest-first: // // 1. the exact (uid, rv) fact, the only exact-capable join; -// 2. the last-writer-wins fact for that uid, for a removal whose rv never matches; -// 3. a collection fact whose uid set contains this object; +// 2. a collection fact whose uid set contains this object; +// 3. the last-writer-wins fact for that uid, for a removal whose rv never matches; // 4. a collection fact whose scope, selector, and window cover it; -// 5. the rv-only escape hatch. +// 5. the rv-only escape hatch; +// 6. the (namespace, name) floor. +// +// The name tier is last because it is the weakest per-object evidence here: a name is reused after a +// delete and recreate, so it can name the author of a previous object that held it, where a uid +// cannot and an rv identifies one specific write. Nothing that carries a uid or an rv ever reaches +// it, so ranking it last costs the stronger tiers nothing and only picks up what they cannot express. // -// Precedence is the correctness argument for the collection tiers. A scope match is the weakest -// evidence here and can name the wrong human, so it is only ever reached when nothing more specific -// applies: an unrelated delete by another actor during the same window is claimed by its own fact -// at tier 2 and never reaches tier 4. +// Precedence is the correctness argument for the collection tiers, and the two of them sit on +// OPPOSITE sides of the latest tier on purpose. +// +// Uid membership outranks it because the two tiers answer different questions. The latest tier says +// who last WROTE an object; a removal asks who DELETED it. For a single-object delete those coincide, +// because the delete files its own fact under that uid — but a collection delete files one fact about +// the collection, so the uid's latest entry is left holding whoever happened to write the object last. +// Ranking it above the collection's uid set credited a removal to the previous editor and never +// reached the actor who actually ran the delete, which is the one thing the deleted expander did get +// right: it overwrote that entry per object. Uid membership is the API server stating that THIS +// request deleted THIS object, so nothing weaker may answer ahead of it. +// +// Scope matching stays below, because it is the weakest evidence here and can name the wrong human: +// an unrelated delete by another actor during the same window is claimed by its own fact at tier 3 +// and never reaches tier 4. func (i *FactIndex) Lookup(query FactQuery) AuthorResolution { now := time.Now() cutoff := now.Add(-i.ttl) @@ -226,32 +339,88 @@ func (i *FactIndex) Lookup(query FactQuery) AuthorResolution { if query.UID != "" && query.ResourceVersion != "" { if fact, found := facts.lookupExact(query.UID, query.ResourceVersion, cutoff); found { - return AuthorResolution{Fact: fact, Result: attributionResultForFact(fact, false)} + return AuthorResolution{Fact: fact, Result: AttributionExact} } } if !query.ExactCapable { - if query.UID != "" { - if fact, found := facts.lookupLatest(query.UID, cutoff); found { - return AuthorResolution{Fact: fact, Result: attributionResultForFact(fact, true)} - } - } - resolution := facts.matchCollection(query, now, cutoff, i.collectionWindow) - if resolution.Result != AttributionAbsent { + if resolution := i.lookupRemoval(facts, query, now, cutoff); resolution.Result != AttributionAbsent { return resolution } } if query.ResourceVersion != "" { if fact, found := facts.lookupRV(query.ResourceVersion, cutoff); found { - return AuthorResolution{Fact: fact, Result: attributionResultForFact(fact, true)} + return AuthorResolution{Fact: fact, Result: AttributionResourceVersion} + } + } + if query.Name != "" { + if fact, found := facts.lookupName(query.Namespace, query.Name, cutoff); found { + return AuthorResolution{Fact: fact, Result: AttributionName} } } return AuthorResolution{Result: AttributionAbsent} } +// lookupRemoval reads the tiers only a removal may consult, in the order they rank: the uid set of +// a collection that named this object, the object's own delete fact (by uid, then by name), its +// last-writer fact, and finally a collection whose scope covers it. See Lookup for why uid +// membership and scope matching sit on opposite sides of the latest tier. +// +// The ordering rule inside it is the one the whole removal path turns on: a fact about the DELETION +// outranks a fact about a write, whichever key each happens to be filed under. A write fact answers +// "who last edited this", which is not the question a removal asks. +func (i *FactIndex) lookupRemoval( + facts *scopeFacts, + query FactQuery, + now, cutoff time.Time, +) AuthorResolution { + var writeFallback AuthorResolution + haveWriteFallback := false + if query.UID != "" { + if fact, found := facts.matchCollectionUID(query, cutoff); found { + return AuthorResolution{Fact: fact, Result: AttributionCollectionUID} + } + if fact, found := facts.lookupLatest(query.UID, cutoff); found { + resolution := AuthorResolution{Fact: fact, Result: AttributionLatest} + // The object's own delete fact: the strongest thing a removal can hope for below uid + // membership, so it ends the search here. + if isRemovalVerb(fact.Verb) { + return resolution + } + // A WRITE fact says who last edited the object, not who deleted it. Hold it and keep + // looking for evidence about the deletion rather than answering with it. + writeFallback, haveWriteFallback = resolution, true + } + } + // The object's own delete fact again, this time keyed by NAME, which is the only key it has when + // the API server answered the delete with a Status rather than the object: there is then no uid + // to recover from the body (measured in corpus configmap/owner-ref-cascade, where the parent's + // delete returns Status, against configmap/finalizer-delete, where it returns the ConfigMap). + // + // It has to be reachable HERE, above the write fallback, or it is not reachable at all for a + // removal: returning the uid tier's write fact ends the lookup, and the caller then holds that + // fact and waits out the whole grace for delete evidence that was sitting in this tier the entire + // time. That wait is not free — it blocks the watch shard's serial goroutine, so every later + // event for the type waits behind it. + if query.Name != "" { + if fact, found := facts.lookupName(query.Namespace, query.Name, cutoff); found && isRemovalVerb(fact.Verb) { + return AuthorResolution{Fact: fact, Result: AttributionName} + } + } + if haveWriteFallback { + return writeFallback + } + if fact, found := facts.matchCollectionScope(query, now, cutoff, i.collectionWindow); found { + return AuthorResolution{Fact: fact, Result: AttributionCollectionScope} + } + return AuthorResolution{Result: AttributionAbsent} +} + // Run follows the subscription set until the context ends, applying what it reads and reporting // what it lost. It returns only when the context ends: a transport failure is retried, because a // follower that gave up would leave attribution silently dead for the life of the process. func (i *FactIndex) Run(ctx context.Context, follower FactFollower) error { + transport := follower.TransportKind() + recordTransportInfo(ctx, transport) subscription := follower.FollowFacts(i.streams.Keys(), i.ttl) i.streams.Observe(subscription.SetStreams) defer i.streams.Observe(nil) @@ -263,12 +432,17 @@ func (i *FactIndex) Run(ctx context.Context, follower FactFollower) error { if ctx.Err() != nil { return nil } + recordFollowerError(ctx, transport) i.log.Error(err, "attribution fact follower failed; retrying") if waitErr := waitBlock(ctx, factFollowErrorBackoff); waitErr != nil { return nil } continue } + // An idle round counts as success: the question the gauge answers is whether the follower is + // READING, and a block period that elapsed with nothing on any stream is a healthy read. Only + // advancing it on a non-empty delivery would make a quiet cluster look like a wedged follower. + recordFollowerSuccess(ctx) for _, entry := range delivery.Entries { i.Apply(ctx, entry) } @@ -276,6 +450,7 @@ func (i *FactIndex) Run(ctx context.Context, follower FactFollower) error { if now := time.Now(); now.Sub(lastSweep) >= i.sweepInterval { i.Sweep(now) lastSweep = now + i.recordSize(ctx) } } } @@ -352,9 +527,21 @@ func (i *FactIndex) file(facts *scopeFacts, scope factScope, fact AuthorFact, no case fact.ResourceVersion != "": facts.putRV(fact.ResourceVersion, &indexedFact{fact: fact, at: now, seq: i.nextSeq()}) return []factWaiterKey{{scope: scope, kind: factKindRV, value: fact.ResourceVersion}} + case fact.Name != "": + // The floor: no uid and no resourceVersion, so only the name can reach it. This is the + // aggregated-API write — the API server proxied the request and never decoded the response, + // so the objectRef holds the name from the URL path and nothing else. Such a fact used to be + // published and then dropped here as unjoinable, which is why an aggregated update or single + // delete shipped committer-authored no matter who ran it. + facts.putName(fact.Namespace, fact.Name, &indexedFact{fact: fact, at: now, seq: i.nextSeq()}) + return []factWaiterKey{{ + scope: scope, kind: factKindName, value: nameWaiterValue(fact.Namespace, fact.Name), + }} default: - // A fact with neither a uid nor a resourceVersion can never be joined. The publish side does - // not produce one; storing it anyway would only fill the index with entries no query reaches. + // A fact with no uid, no resourceVersion and no name can never be joined: nothing about it + // identifies an object. An aggregated CREATE is the case that lands here, because the API + // server assigns the name and the objectRef carries none — though the publish side rejects it + // at the name gate before it reaches this far. return nil } } @@ -462,6 +649,11 @@ func (q FactQuery) waiterKeys() []factWaiterKey { if q.ResourceVersion != "" { keys = append(keys, factWaiterKey{scope: scope, kind: factKindRV, value: q.ResourceVersion}) } + if q.Name != "" { + keys = append(keys, factWaiterKey{ + scope: scope, kind: factKindName, value: nameWaiterValue(q.Namespace, q.Name), + }) + } return keys } @@ -471,6 +663,62 @@ func exactWaiterValue(uid, rv string) string { return uid + "\x00" + rv } +// nameWaiterValue renders the name tier's (namespace, name) pair as one waiter value, on the same +// separator and for the same reason. +func nameWaiterValue(namespace, name string) string { + return namespace + "\x00" + name +} + +// recordSize publishes how much the index holds. It is sampled on the sweep rather than on every +// applied fact: the number is a memory reading, and the sweep is when it has just changed most. +// The v1 gauge cost a Redis SCAN of the whole fact keyspace to produce this; it is now a field read. +func (i *FactIndex) recordSize(ctx context.Context) { + if telemetry.AttributionFactIndexEntries == nil { + return + } + telemetry.AttributionFactIndexEntries.Record(ctx, int64(i.Len())) +} + +// recordFactEvent counts one fact lifecycle event under its bounded op. +func recordFactEvent(ctx context.Context, op string) { + if telemetry.AttributionFactsTotal == nil { + return + } + telemetry.AttributionFactsTotal.Add(ctx, 1, metric.WithAttributes(attribute.String("op", op))) +} + +// recordTransportInfo publishes which transport is carrying the facts, as an info gauge whose value +// is always 1. It is recorded from the follower rather than from the wiring because this is the one +// goroutine that runs for the life of the process and holds the transport. +func recordTransportInfo(ctx context.Context, transport FactTransportKind) { + if telemetry.AttributionTransportInfo == nil { + return + } + telemetry.AttributionTransportInfo.Record(ctx, 1, + metric.WithAttributes(attribute.String("transport", string(transport)))) +} + +// recordFollowerError counts one failed follower read. The follower retries rather than returning, +// so without this the failures are a log line and nothing else. +func recordFollowerError(ctx context.Context, transport FactTransportKind) { + if telemetry.AttributionFactFollowerErrorsTotal == nil { + return + } + telemetry.AttributionFactFollowerErrorsTotal.Add(ctx, 1, + metric.WithAttributes(attribute.String("transport", string(transport)))) +} + +// recordFollowerSuccess stamps the time of the last successful read. It matters more than the error +// counter beside it: a counter says errors are happening, and only this separates "erroring +// occasionally while making progress" from "has read nothing in ten minutes" — the second of which +// is attribution degrading to committer-authored cluster-wide. +func recordFollowerSuccess(ctx context.Context) { + if telemetry.AttributionFactFollowerLastSuccessTimestampSeconds == nil { + return + } + telemetry.AttributionFactFollowerLastSuccessTimestampSeconds.Record(ctx, time.Now().Unix()) +} + // recordFactIndexEviction counts one evicted entry under its bounded reason. func recordFactIndexEviction(ctx context.Context, reason string) { if telemetry.AttributionFactIndexEvictionsTotal == nil { diff --git a/internal/queue/fact_index_store.go b/internal/queue/fact_index_store.go index e8047dbe..48a5b00d 100644 --- a/internal/queue/fact_index_store.go +++ b/internal/queue/fact_index_store.go @@ -8,7 +8,7 @@ import ( "k8s.io/apimachinery/pkg/labels" ) -// factKind names one of the four match structures a fact can land in. The set is closed: a fact +// factKind names one of the five match structures a fact can land in. The set is closed: a fact // that fits none of them is not stored, because a fact nothing can ever join is only memory. type factKind uint8 @@ -19,6 +19,10 @@ const ( factKindLatest // factKindRV is (rv), the escape hatch for a fact with an rv but no uid. factKindRV + // factKindName is (namespace, name), the floor for a fact with neither a uid nor an rv. Only an + // aggregated-API write reaches it today: the API server proxies the request, so the objectRef + // carries the name from the URL and nothing else. + factKindName // factKindCollection is (namespace), time-bounded, serving removals caused by a // deletecollection. factKindCollection @@ -44,6 +48,13 @@ type exactFactKey struct { rv string } +// nameFactKey is the (namespace, name) pair of the name tier. The namespace is part of the key +// because a name is only unique within one, and the scope already binds the type and the route. +type nameFactKey struct { + namespace string + name string +} + // indexedFact is one stored fact plus the insertion time the TTL sweep reads and the sequence // number that tells a stale eviction reference from a live entry. type indexedFact struct { @@ -119,7 +130,10 @@ type factRef struct { kind factKind uid string rv string - seq uint64 + // namespace and name locate a name-tier entry; they are unset for every other kind. + namespace string + name string + seq uint64 } // scopeFacts is one (route, group/resource)'s four match structures, its oldest-first insertion @@ -130,6 +144,7 @@ type scopeFacts struct { exact map[exactFactKey]*indexedFact latest map[string]*indexedFact rvOnly map[string]*indexedFact + byName map[nameFactKey]*indexedFact collections []*indexedCollection // order is every live entry's reference in insertion order, oldest first. It may hold stale // references, which remove skips. @@ -142,6 +157,7 @@ func newScopeFacts() *scopeFacts { exact: map[exactFactKey]*indexedFact{}, latest: map[string]*indexedFact{}, rvOnly: map[string]*indexedFact{}, + byName: map[nameFactKey]*indexedFact{}, } } @@ -175,6 +191,18 @@ func (s *scopeFacts) putRV(rv string, entry *indexedFact) { s.order = append(s.order, factRef{kind: factKindRV, rv: rv, seq: entry.seq}) } +// putName stores the (namespace, name) floor. Like putLatest it is last-writer-wins: two writes to +// one name inside the TTL leave the later actor, which is the same answer the uid tier would give +// for the same pair of writes. +func (s *scopeFacts) putName(namespace, name string, entry *indexedFact) { + key := nameFactKey{namespace: namespace, name: name} + if _, ok := s.byName[key]; !ok { + s.count++ + } + s.byName[key] = entry + s.order = append(s.order, factRef{kind: factKindName, namespace: namespace, name: name, seq: entry.seq}) +} + // putCollection appends one collection fact. Collection facts are not keyed one per namespace: // two actors may delete collections in one namespace within the same window, and each removal must // be able to find the one that covered it. @@ -199,28 +227,48 @@ func (s *scopeFacts) lookupRV(rv string, cutoff time.Time) (AuthorFact, bool) { return liveFact(s.rvOnly[rv], cutoff) } -// matchCollection resolves a removal against the collection tier, in the two passes the design -// orders: uid membership first, because either the API server said it deleted this object or it did -// not, and scope matching second, because it accepts a bounded risk of naming the wrong human and -// so must only ever be reached when nothing more precise applies. -func (s *scopeFacts) matchCollection(q FactQuery, now, cutoff time.Time, window time.Duration) AuthorResolution { +// lookupName reads the (namespace, name) floor. +func (s *scopeFacts) lookupName(namespace, name string, cutoff time.Time) (AuthorFact, bool) { + return liveFact(s.byName[nameFactKey{namespace: namespace, name: name}], cutoff) +} + +// matchCollectionUID resolves a removal against a collection fact that NAMED this object: the API +// server returned the set it deleted, and this uid was in it. There is no over-attribution risk in +// that — either the object was in the set or it was not — which is why it outranks the latest tier. +// +// It carries no window check, deliberately. The uid set is a statement about this exact object +// rather than about a span of time, so the fact's own TTL is the only bound it needs; a window would +// only discard evidence that cannot be wrong. +func (s *scopeFacts) matchCollectionUID(q FactQuery, cutoff time.Time) (AuthorFact, bool) { for i := len(s.collections) - 1; i >= 0; i-- { entry := s.collections[i] if !entry.covers(q, cutoff) || entry.uids == nil { continue } if _, ok := entry.uids[q.UID]; ok { - return AuthorResolution{Fact: entry.fact, Result: AttributionCollectionUID} + return entry.fact, true } } + return AuthorFact{}, false +} + +// matchCollectionScope resolves a removal against a collection fact by scope alone: same type and +// namespace, the request's selector accepting this object's labels, within the collection window. +// It is the weakest evidence the join has and the only tier that can name the wrong human, so it +// runs last — after the object's own facts have all missed. +func (s *scopeFacts) matchCollectionScope( + q FactQuery, + now, cutoff time.Time, + window time.Duration, +) (AuthorFact, bool) { for i := len(s.collections) - 1; i >= 0; i-- { entry := s.collections[i] if !entry.covers(q, cutoff) || !entry.inWindow(now, window) || !entry.selects(q.Labels) { continue } - return AuthorResolution{Fact: entry.fact, Result: AttributionCollectionScope} + return entry.fact, true } - return AuthorResolution{Result: AttributionAbsent} + return AuthorFact{}, false } // sweep drops every entry inserted before the cutoff and reports how many went. It also compacts @@ -245,6 +293,12 @@ func (s *scopeFacts) sweep(cutoff time.Time) int { removed++ } } + for key, entry := range s.byName { + if entry.at.Before(cutoff) { + delete(s.byName, key) + removed++ + } + } kept := s.collections[:0] for _, entry := range s.collections { if entry.at.Before(cutoff) { @@ -286,27 +340,36 @@ func (s *scopeFacts) evictOldest() bool { func (s *scopeFacts) remove(ref factRef) bool { switch ref.kind { case factKindExact: - key := exactFactKey{uid: ref.uid, rv: ref.rv} - if entry, ok := s.exact[key]; ok && entry.seq == ref.seq { - delete(s.exact, key) - return true - } + return removeKeyed(s.exact, exactFactKey{uid: ref.uid, rv: ref.rv}, ref.seq) case factKindLatest: - if entry, ok := s.latest[ref.uid]; ok && entry.seq == ref.seq { - delete(s.latest, ref.uid) - return true - } + return removeKeyed(s.latest, ref.uid, ref.seq) case factKindRV: - if entry, ok := s.rvOnly[ref.rv]; ok && entry.seq == ref.seq { - delete(s.rvOnly, ref.rv) - return true - } + return removeKeyed(s.rvOnly, ref.rv, ref.seq) + case factKindName: + return removeKeyed(s.byName, nameFactKey{namespace: ref.namespace, name: ref.name}, ref.seq) case factKindCollection: - for i, entry := range s.collections { - if entry.seq == ref.seq { - s.collections = append(s.collections[:i], s.collections[i+1:]...) - return true - } + return s.removeCollection(ref.seq) + } + return false +} + +// removeKeyed deletes one map entry when it is still the one the reference was taken for, which is +// what the sequence check decides. It is generic over the key because the four keyed structures +// differ only in how they are addressed. +func removeKeyed[K comparable](m map[K]*indexedFact, key K, seq uint64) bool { + if entry, ok := m[key]; ok && entry.seq == seq { + delete(m, key) + return true + } + return false +} + +// removeCollection drops the collection entry with this sequence, reporting whether it was there. +func (s *scopeFacts) removeCollection(seq uint64) bool { + for i, entry := range s.collections { + if entry.seq == seq { + s.collections = append(s.collections[:i], s.collections[i+1:]...) + return true } } return false @@ -328,14 +391,13 @@ func (s *scopeFacts) compact() { func (s *scopeFacts) live(ref factRef) bool { switch ref.kind { case factKindExact: - entry, ok := s.exact[exactFactKey{uid: ref.uid, rv: ref.rv}] - return ok && entry.seq == ref.seq + return liveKeyed(s.exact, exactFactKey{uid: ref.uid, rv: ref.rv}, ref.seq) case factKindLatest: - entry, ok := s.latest[ref.uid] - return ok && entry.seq == ref.seq + return liveKeyed(s.latest, ref.uid, ref.seq) case factKindRV: - entry, ok := s.rvOnly[ref.rv] - return ok && entry.seq == ref.seq + return liveKeyed(s.rvOnly, ref.rv, ref.seq) + case factKindName: + return liveKeyed(s.byName, nameFactKey{namespace: ref.namespace, name: ref.name}, ref.seq) case factKindCollection: for _, entry := range s.collections { if entry.seq == ref.seq { @@ -346,6 +408,12 @@ func (s *scopeFacts) live(ref factRef) bool { return false } +// liveKeyed reports whether a map still holds the entry a reference was taken for. +func liveKeyed[K comparable](m map[K]*indexedFact, key K, seq uint64) bool { + entry, ok := m[key] + return ok && entry.seq == seq +} + // covers reports whether a collection fact is in scope for a query at all: same namespace, and not // yet aged out. The namespace is part of the collection key, so it binds in both passes — uid // membership included, because the uid set of a collection in another namespace has no business diff --git a/internal/queue/fact_index_test.go b/internal/queue/fact_index_test.go index 5abcd4de..344a81e1 100644 --- a/internal/queue/fact_index_test.go +++ b/internal/queue/fact_index_test.go @@ -3,6 +3,9 @@ package queue import ( + "context" + "errors" + "sync/atomic" "testing" "time" @@ -101,9 +104,7 @@ const factIndexTestUID = "uid-1" // objectFact is one ordinary write's fact. func objectFact(author, rv string) AuthorFact { return AuthorFact{ - GroupResource: "apps/deployments", Namespace: "team-a", - Name: "web", UID: factIndexTestUID, ResourceVersion: rv, Author: author, @@ -115,7 +116,6 @@ func objectFact(author, rv string) AuthorFact { // uids the API server said it covered. func aliceCollectionFact(selector string, uids ...string) AuthorFact { return AuthorFact{ - GroupResource: "apps/deployments", Namespace: "team-a", Author: "alice", Verb: "deletecollection", @@ -143,7 +143,7 @@ func TestFactIndex_JoinPolicyDependsOnTheEventKind(t *testing.T) { // ADDED / MODIFIED present the resourceVersion the write produced, so they join exactly. exact := harness.resolve(objectQuery("prod-eu-1", "uid-1", "101", true)) - require.Equal(t, AttributionExactUser, exact.Result) + require.Equal(t, AttributionExact, exact.Result) require.Equal(t, "alice", exact.Fact.Author) harness.waitForFacts(2) @@ -154,7 +154,7 @@ func TestFactIndex_JoinPolicyDependsOnTheEventKind(t *testing.T) { // A removal's rv never matches the write's, so it is the event kind that consults latest. removal := harness.resolve(objectQuery("prod-eu-1", "uid-1", "999", false)) - require.Equal(t, AttributionWeak, removal.Result) + require.Equal(t, AttributionLatest, removal.Result) require.Equal(t, "alice", removal.Fact.Author) } @@ -166,7 +166,7 @@ func TestFactIndex_RouteIsolatesOtherwiseIdenticalFacts(t *testing.T) { harness.publish(factIndexTestStream("prod-eu-1"), objectFact("alice", "101")) harness.publish(factIndexTestStream("prod-us-1"), objectFact("bob", "101")) // And the rv-only hatch, which is where it bites hardest because it carries no uid at all. - rvOnly := AuthorFact{GroupResource: "apps/deployments", Namespace: "team-a", ResourceVersion: "202", Verb: "update"} + rvOnly := AuthorFact{Namespace: "team-a", ResourceVersion: "202", Verb: "update"} euOnly, usOnly := rvOnly, rvOnly euOnly.Author, usOnly.Author = "eu-rv", "us-rv" harness.publish(factIndexTestStream("prod-eu-1"), euOnly) @@ -218,7 +218,7 @@ func TestFactIndex_PerTypeCapEvictsOldestFirstAndCountsIt(t *testing.T) { // rv-only facts occupy one entry each, so the cap counts what the test publishes. harness := newFactIndexHarness(t, FactIndexConfig{MaxFactsPerType: 2}) rvFact := func(author, rv string) AuthorFact { - return AuthorFact{GroupResource: "apps/deployments", ResourceVersion: rv, Author: author, Verb: "update"} + return AuthorFact{ResourceVersion: rv, Author: author, Verb: "update"} } key := factIndexTestStream("prod-eu-1") harness.publish(key, rvFact("first", "1"), rvFact("second", "2"), rvFact("third", "3")) @@ -227,7 +227,7 @@ func TestFactIndex_PerTypeCapEvictsOldestFirstAndCountsIt(t *testing.T) { other := FactStreamKeyFor("prod-eu-1", schema.GroupResource{Resource: "configmaps"}) harness.publish( other, - AuthorFact{GroupResource: "configmaps", ResourceVersion: "9", Author: "quiet", Verb: "update"}, + AuthorFact{ResourceVersion: "9", Author: "quiet", Verb: "update"}, ) quiet := FactQuery{AuditRoute: "prod-eu-1", GroupResource: schema.GroupResource{Resource: "configmaps"}, @@ -251,14 +251,14 @@ func TestFactIndex_TotalCapEvictsFromTheLargestType(t *testing.T) { harness := newFactIndexHarness(t, FactIndexConfig{MaxFactsTotal: 2}) busy := factIndexTestStream("prod-eu-1") harness.publish(busy, - AuthorFact{GroupResource: "apps/deployments", ResourceVersion: "1", Author: "first", Verb: "update"}, - AuthorFact{GroupResource: "apps/deployments", ResourceVersion: "2", Author: "second", Verb: "update"}, + AuthorFact{ResourceVersion: "1", Author: "first", Verb: "update"}, + AuthorFact{ResourceVersion: "2", Author: "second", Verb: "update"}, ) require.Equal(t, "second", harness.resolve(objectQuery("prod-eu-1", "", "2", true)).Fact.Author) quiet := FactStreamKeyFor("prod-eu-1", schema.GroupResource{Resource: "configmaps"}) harness.publish(quiet, - AuthorFact{GroupResource: "configmaps", ResourceVersion: "9", Author: "quiet", Verb: "update"}) + AuthorFact{ResourceVersion: "9", Author: "quiet", Verb: "update"}) // The overflow falls on the type holding the most, so the pressure lands where it came from. quietQuery := FactQuery{AuditRoute: "prod-eu-1", GroupResource: schema.GroupResource{Resource: "configmaps"}, @@ -391,24 +391,58 @@ func TestFactIndex_CollectionScopeMatchStopsAtTheWindow(t *testing.T) { harness.absent(objectQuery("prod-eu-1", "uid-9", "9", false)) } -func TestFactIndex_CollectionIsWeakerThanAnObjectsOwnFact(t *testing.T) { +// A collection fact that named no uids is evidence about a SCOPE, so it must not outrank an +// object's own fact. This is the precedence that keeps scope matching safe: an unrelated delete by +// another actor during the same window is claimed by its own fact and never reaches the scope tier. +func TestFactIndex_CollectionScopeIsWeakerThanAnObjectsOwnFact(t *testing.T) { harness := newFactIndexHarness(t, FactIndexConfig{}) key := factIndexTestStream("prod-eu-1") harness.publish(key, aliceCollectionFact(""), objectFact("bob", "101")) harness.waitForFacts(3) - // Precedence is the correctness argument: an unrelated delete by another actor during the same - // window is claimed by its own fact and never reaches the scope tier. resolution := harness.resolve(objectQuery("prod-eu-1", "uid-1", "999", false)) - require.Equal(t, AttributionWeak, resolution.Result) + require.Equal(t, AttributionLatest, resolution.Result) require.Equal(t, "bob", resolution.Fact.Author) } +// TestFactIndex_CollectionUIDOutranksAStaleWriteFact is the other side of that precedence, and it +// is the one the tiers originally got wrong. +// +// The latest tier says who last WROTE an object; a removal asks who DELETED it. For a single-object +// delete the two coincide, because the delete files its own fact under that uid. A COLLECTION delete +// files one fact about the collection instead, so the uid's latest entry is left holding whoever +// edited the object last — and ranking it above the collection's uid set credited every removal to +// the previous editor, never reaching the actor who ran the delete. That is the one thing the +// deleted expander got right, by overwriting that entry per object. +// +// The e2e specs could not catch it: they create and delete as the same actor, so the wrong answer +// and the right answer are the same name. +func TestFactIndex_CollectionUIDOutranksAStaleWriteFact(t *testing.T) { + harness := newFactIndexHarness(t, FactIndexConfig{}) + key := factIndexTestStream("prod-eu-1") + + // Bob edits the object; alice then deletes the collection that covers it, and the API server + // returns the set, so the fact names this uid. + harness.publish(key, objectFact("bob", "101"), aliceCollectionFact("", factIndexTestUID)) + harness.waitForFacts(3) + + resolution := harness.resolve(objectQuery("prod-eu-1", factIndexTestUID, "999", false)) + require.Equal(t, AttributionCollectionUID, resolution.Result) + require.Equal(t, "alice", resolution.Fact.Author, + "the removal was caused by alice's collection delete, not by bob's earlier update") + + // A create/update on the same object is NOT a removal, so it keeps resolving to its own writer: + // the collection tiers are only ever consulted for an event whose rv cannot match. + write := harness.resolve(objectQuery("prod-eu-1", factIndexTestUID, "101", true)) + require.Equal(t, "bob", write.Fact.Author) +} + func TestFactIndex_UnjoinableFactIsNotStored(t *testing.T) { harness := newFactIndexHarness(t, FactIndexConfig{}) key := factIndexTestStream("prod-eu-1") harness.publish(key, - AuthorFact{GroupResource: "apps/deployments", Author: "nobody", Verb: "update"}, + // No uid, no resourceVersion and no name: nothing about it identifies an object. + AuthorFact{Author: "nobody", Verb: "update"}, objectFact("alice", "101"), ) require.Equal(t, "alice", harness.resolve(objectQuery("prod-eu-1", "uid-1", "101", true)).Fact.Author) @@ -417,6 +451,148 @@ func TestFactIndex_UnjoinableFactIsNotStored(t *testing.T) { require.Equal(t, 2, harness.index.Len()) } +// aggregatedFact is the shape an aggregated-API write produces: the API server proxied the request +// and never decoded the response, so the objectRef carries the name from the URL path and neither a +// uid nor a resourceVersion. Measured in corpus flunder/aggregated-api-delete. +func aggregatedFact(author, name, verb string) AuthorFact { + return AuthorFact{Namespace: "team-a", Name: name, Author: author, Verb: verb} +} + +// namedQuery is a watch event that knows its own name, which every watch event does. +func namedQuery(uid, rv, name string, exactCapable bool) FactQuery { + query := objectQuery("prod-eu-1", uid, rv, exactCapable) + query.Name = name + return query +} + +func TestFactIndex_NameTierJoinsAFactCarryingNoUIDOrResourceVersion(t *testing.T) { + harness := newFactIndexHarness(t, FactIndexConfig{}) + harness.publish(factIndexTestStream("prod-eu-1"), aggregatedFact("alice", "fl-1", "update")) + + // The watch event carries the full object, so it has a uid and an rv the fact does not. Every + // stronger tier therefore misses, and the name is the only thing the two sides share. + resolution := harness.resolve(namedQuery("uid-9", "77", "fl-1", true)) + require.Equal(t, AttributionName, resolution.Result) + require.Equal(t, "alice", resolution.Fact.Author) + + // A different object of the same type in the same namespace is not covered by it. + harness.absent(namedQuery("uid-8", "78", "fl-2", true)) +} + +func TestFactIndex_NameTierIsScopedToItsNamespace(t *testing.T) { + harness := newFactIndexHarness(t, FactIndexConfig{}) + other := aggregatedFact("bob", "fl-1", "update") + other.Namespace = "team-b" + harness.publish(factIndexTestStream("prod-eu-1"), other) + harness.waitForFacts(1) + + // Same type, same route, same name, different namespace: a name is unique only within one. + harness.absent(namedQuery("uid-9", "77", "fl-1", true)) +} + +func TestFactIndex_NameTierRanksBelowEveryStrongerTier(t *testing.T) { + harness := newFactIndexHarness(t, FactIndexConfig{}) + // A name-only fact and a uid-bearing one about the same object. The uid tiers must win: a name is + // reused after a delete and recreate, so it is the weakest per-object evidence there is. + harness.publish(factIndexTestStream("prod-eu-1"), + aggregatedFact("name-tier", "fl-1", "update"), + objectFact("uid-tier", "101"), + ) + harness.waitForFacts(3) + + exact := harness.resolve(namedQuery(factIndexTestUID, "101", "fl-1", true)) + require.Equal(t, AttributionExact, exact.Result) + require.Equal(t, "uid-tier", exact.Fact.Author) + + // And the latest tier, which a removal consults, also outranks it. + removal := harness.resolve(namedQuery(factIndexTestUID, "999", "fl-1", false)) + require.Equal(t, "uid-tier", removal.Fact.Author) +} + +func TestFactIndex_NameTierResolvesAnAggregatedRemovalToItsDeleter(t *testing.T) { + harness := newFactIndexHarness(t, FactIndexConfig{}) + // The whole point of restoring the name: an aggregated single delete is audited with a name and + // nothing else, so before this tier it was published and then dropped, and the removal shipped + // committer-authored however ran it. + harness.publish(factIndexTestStream("prod-eu-1"), aggregatedFact("alice", "fl-del", "delete")) + + removal := harness.resolve(namedQuery("uid-9", "999", "fl-del", false)) + require.Equal(t, AttributionName, removal.Result) + require.Equal(t, "alice", removal.Fact.Author) +} + +func TestFactIndex_ARemovalReachesItsDeleteFactKeyedByName(t *testing.T) { + harness := newFactIndexHarness(t, FactIndexConfig{}) + // The shape a built-in delete produces when the API server answers it with a Status rather than + // the object: there is no uid in the body to recover, so the delete fact is keyed by name alone + // (corpus configmap/owner-ref-cascade). The object also has an ordinary write fact, keyed by uid. + harness.publish(factIndexTestStream("prod-eu-1"), + objectFact("last-editor", "101"), + aggregatedFact("the-deleter", "cm-parent", "delete"), + ) + harness.waitForFacts(3) + + // The removal must reach the delete fact. Answering with the uid tier's write fact would both + // name the wrong actor and, worse, keep the caller waiting out the whole grace for evidence that + // is already here — blocking the watch shard for every later event of this type. + removal := harness.resolve(namedQuery(factIndexTestUID, "999", "cm-parent", false)) + require.Equal(t, AttributionName, removal.Result) + require.Equal(t, "the-deleter", removal.Fact.Author) +} + +func TestFactIndex_ANameKeyedWriteDoesNotOutrankTheObjectsOwnUIDFact(t *testing.T) { + harness := newFactIndexHarness(t, FactIndexConfig{}) + // Only a fact about the DELETION may jump ahead of the uid tier. A name-keyed WRITE is not that, + // and must not displace the object's own uid-keyed evidence. + harness.publish(factIndexTestStream("prod-eu-1"), + objectFact("uid-tier", "101"), + aggregatedFact("name-tier-write", "cm-parent", "update"), + ) + harness.waitForFacts(3) + + removal := harness.resolve(namedQuery(factIndexTestUID, "999", "cm-parent", false)) + require.Equal(t, "uid-tier", removal.Fact.Author) +} + +func TestFactIndex_ARemovalStillPrefersItsOwnUIDKeyedDeleteFact(t *testing.T) { + harness := newFactIndexHarness(t, FactIndexConfig{}) + uidDelete := objectFact("uid-deleter", "102") + uidDelete.Verb = "delete" + harness.publish(factIndexTestStream("prod-eu-1"), + uidDelete, + aggregatedFact("name-deleter", "cm-parent", "delete"), + ) + harness.waitForFacts(3) + + // Both are about the deletion; the uid-keyed one identifies the object exactly, so it wins. + removal := harness.resolve(namedQuery(factIndexTestUID, "999", "cm-parent", false)) + require.Equal(t, "uid-deleter", removal.Fact.Author) +} + +func TestFactIndex_NameFactWakesAWaiterThatArrivedFirst(t *testing.T) { + harness := newFactIndexHarness(t, FactIndexConfig{}) + key := factIndexTestStream("prod-eu-1") + harness.index.Streams().Acquire(key) + + // The watch beats the audit event, which is the ordinary case. The waiter must be registered + // under the name tier too, or the query sleeps out its whole grace beside a fact that would match. + resolved := make(chan AuthorResolution, 1) + go func() { + resolved <- harness.resolve(namedQuery("uid-9", "77", "fl-late", true)) + }() + + require.NoError(t, harness.transport.PublishFacts(t.Context(), key, + []AuthorFact{aggregatedFact("alice", "fl-late", "update")})) + + select { + case resolution := <-resolved: + require.Equal(t, AttributionName, resolution.Result) + require.Equal(t, "alice", resolution.Fact.Author) + case <-time.After(factIndexTestGrace * 2): + t.Fatal("a name-tier fact never woke the waiting query") + } +} + func TestFactIndex_TrimGapIsCountedAndNamed(t *testing.T) { reader, err := telemetry.InitTestExporter() require.NoError(t, err) @@ -481,3 +657,279 @@ func TestFactIndex_FollowerPicksUpATypeWhenAWatchStartsCoveringIt(t *testing.T) defer release() require.Equal(t, "alice", harness.resolve(objectQuery("prod-eu-1", "uid-1", "101", true)).Fact.Author) } + +// TestFactIndex_AFactAgesFromWhenItWasAppended pins the TTL against the delivery path. +// +// The follower replays the whole retention window on start, so an entry appended nine minutes ago +// is read now. Stamping it with the READ time would hand it a second full TTL, and a process that +// restarted every nine minutes would keep facts alive forever — the horizon would bound nothing. +// The same applies to a follower that fell behind, and to any transport that hands back an entry +// its own retention should have dropped. +func TestFactIndex_AFactAgesFromWhenItWasAppended(t *testing.T) { + index := NewFactIndex(FactIndexConfig{TTL: time.Minute}) + key := factIndexTestStream("prod-eu-1") + + // An entry whose position says it was appended two minutes ago, delivered now. + stale := FactEntry{ + Key: key, + ID: streamIDAt(time.Now().Add(-2 * time.Minute)), + Facts: []AuthorFact{objectFact("alice", "101")}, + } + index.Apply(t.Context(), stale) + + require.Equal(t, AttributionAbsent, + index.Lookup(objectQuery("prod-eu-1", factIndexTestUID, "101", true)).Result, + "a fact older than the TTL must not become joinable just because it was read late") + + // The same entry appended now is joinable, so the check is on age and not on the ID's shape. + fresh := FactEntry{Key: key, ID: streamIDAt(time.Now()), Facts: []AuthorFact{objectFact("alice", "101")}} + index.Apply(t.Context(), fresh) + require.Equal(t, "alice", + index.Lookup(objectQuery("prod-eu-1", factIndexTestUID, "101", true)).Fact.Author) +} + +// TestFactIndex_OneFactServesEveryGitTargetWaitingForIt is the fan-in property, which is the reason +// there is one index per process rather than one per GitTarget. +// +// Two GitTargets mirroring the same object each run their own watch shard, so each gets its own +// watch event and resolves independently — but the fact naming the author is about a write that +// happened in Kubernetes, not about who is interested in it. Both therefore resolve from the SAME +// stored fact. Nothing is consumed and nothing competes: waking is a broadcast over the set of +// waiters on a key, so there is no "winner" and no second copy. +func TestFactIndex_OneFactServesEveryGitTargetWaitingForIt(t *testing.T) { + harness := newFactIndexHarness(t, FactIndexConfig{}) + key := factIndexTestStream("prod-eu-1") + query := objectQuery("prod-eu-1", factIndexTestUID, "101", true) + + // Both shards ask before the fact exists, so both park on the waiter registry. + const consumers = 2 + results := make(chan AuthorResolution, consumers) + for range consumers { + go func() { + results <- harness.index.Await(t.Context(), query, 5*time.Second) + }() + } + require.Eventually(t, func() bool { return harness.index.waiters.len() > 0 }, + 2*time.Second, 5*time.Millisecond, "both resolvers must be registered before the fact lands") + + // ONE fact is published, once. + harness.publish(key, objectFact("alice", "101")) + + for range consumers { + select { + case resolution := <-results: + require.Equal(t, "alice", resolution.Fact.Author) + require.Equal(t, AttributionExact, resolution.Result) + case <-time.After(10 * time.Second): + t.Fatal("a waiter was never woken: waking must reach every resolver on the key, not one") + } + } + + // The index holds that one fact, not one per consumer, and every waiter is gone. + require.Equal(t, 2, harness.index.Len(), "one write is one exact entry plus one latest entry") + require.Zero(t, harness.index.waiters.len(), "a resolver must leave nothing registered behind") + + // A third shard arriving after the fact is already stored takes the fast path instead: it + // registers, finds it on the immediate check, and never blocks. + late := harness.index.Await(t.Context(), query, 0) + require.Equal(t, "alice", late.Fact.Author) +} + +// TestFactIndex_ARemovalWaitsForEvidenceAboutTheDeletion is the race the collection-precedence fix +// alone did not close, and the one that made an e2e spec pass at one process and fail at four. +// +// Precedence only decides between facts that are BOTH present. The watch event reliably arrives +// before the audit batch carrying its delete — that is the entire reason the grace window exists — +// so at the moment a removal is resolved, the only fact present is often the object's last WRITE. +// Returning on it answered "who deleted this" with "who last edited it", and no ordering of the +// tiers could have helped, because the right fact had not been delivered yet. +func TestFactIndex_ARemovalWaitsForEvidenceAboutTheDeletion(t *testing.T) { + harness := newFactIndexHarness(t, FactIndexConfig{}) + key := factIndexTestStream("prod-eu-1") + + // Bob edits the object. That is the only fact in the index when the removal is resolved. + harness.publish(key, objectFact("bob", "101")) + harness.waitForFacts(2) + + // Alice's collection delete is still in flight, and lands while the resolver waits. + go func() { + time.Sleep(50 * time.Millisecond) + harness.publish(key, aliceCollectionFact("", factIndexTestUID)) + }() + + resolution := harness.index.Await(t.Context(), + removalFactQuery("prod-eu-1", factIndexTestUID), 5*time.Second) + require.Equal(t, AttributionCollectionUID, resolution.Result) + require.Equal(t, "alice", resolution.Fact.Author, + "a removal must wait for evidence about the deletion, not settle for the last edit") +} + +// Waiting must never cost an attribution. When nothing better arrives, the write fact that was held +// back is returned exactly as it would have been returned immediately — the only difference is the +// wait, and a removal that spends its grace is the case the grace window is for. +func TestFactIndex_ARemovalStillNamesTheLastWriterWhenNothingBetterArrives(t *testing.T) { + harness := newFactIndexHarness(t, FactIndexConfig{}) + harness.publish(factIndexTestStream("prod-eu-1"), objectFact("bob", "101")) + harness.waitForFacts(2) + + start := time.Now() + resolution := harness.index.Await(t.Context(), + removalFactQuery("prod-eu-1", factIndexTestUID), 150*time.Millisecond) + + require.Equal(t, AttributionLatest, resolution.Result) + require.Equal(t, "bob", resolution.Fact.Author, "waiting must not turn a match into an absence") + require.GreaterOrEqual(t, time.Since(start), 150*time.Millisecond, + "the fallback is only taken once the grace has actually elapsed") +} + +// An object's OWN delete fact is the strongest evidence a removal can have, so it ends the wait +// immediately rather than holding out for a collection fact that may never come. +func TestFactIndex_ARemovalsOwnDeleteFactEndsTheWaitAtOnce(t *testing.T) { + harness := newFactIndexHarness(t, FactIndexConfig{}) + deleteFact := objectFact("carol", "") + deleteFact.Verb = "delete" + harness.publish(factIndexTestStream("prod-eu-1"), deleteFact) + harness.waitForFacts(1) + + start := time.Now() + resolution := harness.index.Await(t.Context(), + removalFactQuery("prod-eu-1", factIndexTestUID), 5*time.Second) + + require.Equal(t, "carol", resolution.Fact.Author) + require.Less(t, time.Since(start), 2*time.Second, "a delete fact must not be held as a fallback") +} + +// removalFactQuery is a DELETE watch event: its resourceVersion is never the one a write produced. +func removalFactQuery(route, uid string) FactQuery { + return FactQuery{ + AuditRoute: route, + GroupResource: deploymentsGroupResource(), + UID: uid, + ResourceVersion: "999", + Namespace: "team-a", + ExactCapable: false, + } +} + +// TestFactIndex_LatestAndResourceVersionAreDistinctTiers pins the split the metric surface turns +// on. Both used to report "weak", and they are different evidence: the uid tier is the OBJECT's own +// last write, while the rv-only hatch is a fact that carried a resourceVersion and no uid at all. +// The removal path turns on the first specifically, so a reader that cannot tell them apart cannot +// tell a held fallback from an unidentified match. +func TestFactIndex_LatestAndResourceVersionAreDistinctTiers(t *testing.T) { + harness := newFactIndexHarness(t, FactIndexConfig{}) + key := factIndexTestStream("prod-eu-1") + rvOnly := AuthorFact{Namespace: "team-a", ResourceVersion: "202", Author: "rv-actor", Verb: "update"} + harness.publish(key, objectFact("alice", "101"), rvOnly) + harness.waitForFacts(3) + + // A removal joins the object's own uid: the latest tier. + removal := harness.resolve(objectQuery("prod-eu-1", factIndexTestUID, "999", false)) + require.Equal(t, AttributionLatest, removal.Result) + require.Equal(t, "alice", removal.Fact.Author) + + // A fact with no uid is reachable only through the resourceVersion it carried. + hatch := harness.resolve(objectQuery("prod-eu-1", "", "202", true)) + require.Equal(t, AttributionResourceVersion, hatch.Result) + require.Equal(t, "rv-actor", hatch.Fact.Author) +} + +// TestFactIndex_ActorKindIsDerivedFromTheAuthor covers the second half of the label split: the tier +// says which evidence answered, the actor kind says who it named, and every tier can name either. +func TestFactIndex_ActorKindIsDerivedFromTheAuthor(t *testing.T) { + require.Equal(t, ActorKindUser, AuthorFact{Author: "alice"}.ActorKind()) + require.Equal(t, ActorKindServiceAccount, + AuthorFact{Author: "system:serviceaccount:flux-system:kustomize-controller"}.ActorKind()) + require.Equal(t, ActorKindNone, AuthorFact{}.ActorKind()) + + // A resolution that matched nothing names nobody, whatever fact it is carrying. + require.Equal(t, ActorKindNone, AuthorResolution{Result: AttributionAbsent}.ActorKind()) + require.Equal(t, ActorKindUser, + AuthorResolution{Result: AttributionName, Fact: AuthorFact{Author: "alice"}}.ActorKind()) +} + +// erroringFollower fails every read, which is the shape of a wedged follower: Run does not give up +// on a transport error, so without a counter and a timestamp the failure is a log line and a slowly +// rising unresolved rate with nothing pointing at the cause. +type erroringFollower struct { + kind FactTransportKind + // failures counts the reads that have failed, so a test can wait for the retry loop to turn. + failures atomic.Int64 +} + +func (f *erroringFollower) FollowFacts([]FactStreamKey, time.Duration) FactSubscription { + return f +} + +func (f *erroringFollower) TransportKind() FactTransportKind { return f.kind } + +func (f *erroringFollower) SetStreams([]FactStreamKey) {} + +func (f *erroringFollower) Next(ctx context.Context) (FactDelivery, error) { + if err := ctx.Err(); err != nil { + return FactDelivery{}, err + } + f.failures.Add(1) + return FactDelivery{}, errors.New("transport is wedged") +} + +func TestFactIndex_FollowerErrorsAreCountedAndTheTransportIsNamed(t *testing.T) { + reader, err := telemetry.InitTestExporter() + require.NoError(t, err) + + index := NewFactIndex(FactIndexConfig{}) + follower := &erroringFollower{kind: FactTransportRedis} + ctx, cancel := context.WithCancel(t.Context()) + done := make(chan struct{}) + var runErr error + go func() { + defer close(done) + runErr = index.Run(ctx, follower) + }() + require.Eventually(t, func() bool { return follower.failures.Load() >= 1 }, + 5*time.Second, 10*time.Millisecond) + cancel() + <-done + require.NoError(t, runErr, "a transport error is retried, never returned") + + errorCount, found := telemetry.CollectInt64Sum(reader, + "gitopsreverser_attribution_fact_follower_errors_total", + map[string]string{"transport": string(FactTransportRedis)}) + require.True(t, found) + require.Positive(t, errorCount) + + // The info gauge is recorded by the follower, so it is in force for exactly as long as the + // process is reading facts, and it says which contract the other metrics are read under. + info, found := telemetry.CollectInt64Sum(reader, "gitopsreverser_attribution_transport_info", + map[string]string{"transport": string(FactTransportRedis)}) + require.True(t, found) + require.Equal(t, int64(1), info) + + // A follower that has only ever errored has never succeeded, so the liveness gauge must not + // claim it has: this is the difference between "erroring while progressing" and an outage. + _, succeeded := telemetry.CollectInt64Sum(reader, + "gitopsreverser_attribution_fact_follower_last_success_timestamp_seconds", nil) + require.False(t, succeeded) +} + +// A successful read stamps the liveness gauge, idle rounds included: the question it answers is +// whether the follower is READING, and a quiet cluster must not look like a wedged follower. +func TestFactIndex_FollowerSuccessStampsTheLivenessGauge(t *testing.T) { + reader, err := telemetry.InitTestExporter() + require.NoError(t, err) + + before := time.Now().Unix() + harness := newFactIndexHarness(t, FactIndexConfig{}) + harness.publish(factIndexTestStream("prod-eu-1"), objectFact("alice", "101")) + harness.waitForFacts(2) + + stamp, found := telemetry.CollectInt64Sum(reader, + "gitopsreverser_attribution_fact_follower_last_success_timestamp_seconds", nil) + require.True(t, found) + require.GreaterOrEqual(t, stamp, before) + + info, found := telemetry.CollectInt64Sum(reader, "gitopsreverser_attribution_transport_info", + map[string]string{"transport": string(FactTransportMemory)}) + require.True(t, found) + require.Equal(t, int64(1), info) +} diff --git a/internal/queue/fact_stream.go b/internal/queue/fact_stream.go index e475cd1d..d8b75335 100644 --- a/internal/queue/fact_stream.go +++ b/internal/queue/fact_stream.go @@ -12,7 +12,12 @@ import ( "sync" "time" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" "k8s.io/apimachinery/pkg/runtime/schema" + logf "sigs.k8s.io/controller-runtime/pkg/log" + + "github.com/ConfigButler/gitops-reverser/internal/telemetry" ) // Defaults for the attribution fact transport. They are shared by both implementations so a @@ -122,6 +127,21 @@ type FactPublisher interface { PublishFacts(ctx context.Context, key FactStreamKey, facts []AuthorFact) error } +// FactTransportKind is the bounded name of the transport carrying the facts. It is metric metadata +// rather than behavior: nothing above the seam branches on it, but every reading of the attribution +// metrics depends on it, because the two transports fail differently — a burst of unresolved +// commits after a restart is expected under memory, which drops every fact with the process, and a +// bug under redis. +type FactTransportKind string + +const ( + // FactTransportRedis is the Redis Streams transport, the default and the only multi-replica one. + FactTransportRedis FactTransportKind = "redis" + // FactTransportMemory is the in-process ring, which requires a single replica and loses every + // fact on restart by design. + FactTransportMemory FactTransportKind = "memory" +) + // FactFollower follows a set of streams from a horizon. The fact index is its only caller: it // follows the union of the types any watch covers and applies what it reads into memory. type FactFollower interface { @@ -130,6 +150,11 @@ type FactFollower interface { // window before the first watch event needs it. Entries older than the horizon are skipped, // to within the millisecond granularity of a stream position. FollowFacts(keys []FactStreamKey, horizon time.Duration) FactSubscription + + // TransportKind names this transport for the metric labels. It is on the follower half rather + // than beside the wiring because the index is what records follower health, and a counter that + // cannot say which transport erred says half of what an operator needs. + TransportKind() FactTransportKind } // FactSubscription is one follower's live position across its followed streams. It is not safe to @@ -177,6 +202,30 @@ func decodeFactBatch(raw []byte) ([]AuthorFact, error) { return facts, nil } +// recordFactStreamDecodeError counts and says out loud that one entry could not be decoded. +// +// Both transports skip such an entry and advance past it, which is the right call — it can never +// decode, and stalling the follower on it would cost every later fact on that stream — but it makes +// this the one loss path with NO other symptom. A trim gap is detectable after the fact and a +// publish failure is retried by the API server; a skipped entry simply never existed, and the +// commits that needed its facts are authored unresolved with nothing pointing at why. +func recordFactStreamDecodeError( + ctx context.Context, + transport FactTransportKind, + key FactStreamKey, + id string, + err error, +) { + if telemetry.AttributionFactStreamDecodeErrorsTotal != nil { + telemetry.AttributionFactStreamDecodeErrorsTotal.Add(ctx, 1, + metric.WithAttributes(attribute.String("transport", string(transport)))) + } + logf.Log.WithName("attribution-fact-stream").Error(err, + "attribution fact stream entry could not be decoded; it is skipped and its facts are lost, so "+ + "the commits that needed them are authored unresolved", + "transport", string(transport), "stream", key.String(), "entry", id) +} + // streamIDAt renders the position a stream reads from for entries appended at or after t. Stream // IDs are millisecond timestamps, so a time horizon is a plain position and needs no side index. // The position is EXCLUSIVE — a follower reads entries strictly after it — which is why an entry @@ -311,6 +360,23 @@ func (f *followSet) advance(key FactStreamKey, cursor string, behind bool) { } } +// caughtUp clears one stream's behind mark without moving its cursor. A follower that asked a +// stream for entries and was given none has, by definition, read everything there is. +// +// It exists because behind is otherwise STICKY: it is set when a read fills its entry budget, and a +// read that fills the budget exactly is indistinguishable from one that left more waiting. Without +// this the mark survives every later empty read, so a caught-up follower stays flagged until the +// next non-empty one — and if ordinary retention ages out the entries it already read in the +// meantime, trim-gap detection reports a data loss that never happened. The precondition is meant +// to mean "was actually behind", so it has to be cleared by the evidence that it is not. +func (f *followSet) caughtUp(key FactStreamKey) { + f.mu.Lock() + defer f.mu.Unlock() + if state, ok := f.states[key]; ok { + state.behind = false + } +} + // noteGap records a trim gap and reports whether it is new. The same gap is reported once, not on // every pass until the follower catches up past it. func (f *followSet) noteGap(key FactStreamKey, firstSurviving string) bool { diff --git a/internal/queue/fact_stream_conformance_test.go b/internal/queue/fact_stream_conformance_test.go index cf580c88..291e3e25 100644 --- a/internal/queue/fact_stream_conformance_test.go +++ b/internal/queue/fact_stream_conformance_test.go @@ -8,8 +8,11 @@ import ( "testing" "time" + "github.com/redis/go-redis/v9" "github.com/stretchr/testify/require" "k8s.io/apimachinery/pkg/runtime/schema" + + "github.com/ConfigButler/gitops-reverser/internal/telemetry" ) // The attribution fact transport has two implementations and ONE test suite. They are the same @@ -56,10 +59,14 @@ func defaultFactStreamParams() factStreamParams { } } -// factStreamImplementation names one transport and builds it for one case. +// factStreamImplementation names one transport and builds it for one case. appendRaw writes an +// entry payload verbatim, which is the only thing a transport's own internals are needed for here: +// PublishFacts cannot produce an entry that violates the wire contract, and refusing one is the +// loss path with no other symptom. type factStreamImplementation struct { - name string - build func(t *testing.T, params factStreamParams) FactTransport + name string + build func(t *testing.T, params factStreamParams) FactTransport + appendRaw func(t *testing.T, transport FactTransport, key FactStreamKey, payload string) } func factStreamImplementations() []factStreamImplementation { @@ -79,6 +86,15 @@ func factStreamImplementations() []factStreamImplementation { ReadCount: params.ReadCount, }) }, + appendRaw: func(t *testing.T, transport FactTransport, key FactStreamKey, payload string) { + t.Helper() + stream, ok := transport.(*RedisFactStream) + require.True(t, ok) + require.NoError(t, stream.client.XAdd(t.Context(), &redis.XAddArgs{ + Stream: stream.streamKey(key), + Values: map[string]any{factStreamEntryField: payload}, + }).Err()) + }, }, { name: "memory", @@ -91,6 +107,16 @@ func factStreamImplementations() []factStreamImplementation { ReadCount: params.ReadCount, }) }, + appendRaw: func(t *testing.T, transport FactTransport, key FactStreamKey, payload string) { + t.Helper() + stream, ok := transport.(*MemoryFactStream) + require.True(t, ok) + stream.mu.Lock() + defer stream.mu.Unlock() + ring, ok := stream.streams[key] + require.True(t, ok, "publish something before writing a raw entry") + ring.append([]byte(payload), time.Now()) + }, }, } } @@ -189,6 +215,52 @@ func TestFactStreamConformance_AppendsAreFollowedInOrder(t *testing.T) { }) } +// TestFactStreamConformance_ARefusedEntryIsSkippedAndCounted pins the loss path that had no +// symptom at all, over both ways an entry can be refused: a payload that is not JSON, and one that +// is JSON but violates the fact contract by naming nobody. +// +// A refused entry is skipped and its position passed — which is right, since it can never decode and +// stalling on it would cost every later fact on that stream — so the facts it carried are gone. +// Unlike a trim gap that is not detectable after the fact, and unlike a publish failure the API +// server does not retry it, which is why the counter IS the symptom. +func TestFactStreamConformance_ARefusedEntryIsSkippedAndCounted(t *testing.T) { + payloads := map[string]string{ + "malformed": "{not json", + "authorless fact": `[{"uid":"uid-1","resourceVersion":"101","verb":"update"}]`, + "null author": `[{"uid":"uid-1","author":null,"verb":"update"}]`, + } + for name, payload := range payloads { + t.Run(name, func(t *testing.T) { + for _, impl := range factStreamImplementations() { + t.Run(impl.name, func(t *testing.T) { + reader, err := telemetry.InitTestExporter() + require.NoError(t, err) + + transport := impl.build(t, defaultFactStreamParams()) + ctx := t.Context() + key := factStreamTestKey("", "configmaps") + sub := transport.FollowFacts([]FactStreamKey{key}, factStreamTestHorizon) + + require.NoError(t, transport.PublishFacts(ctx, key, authorFacts("alice"))) + impl.appendRaw(t, transport, key, payload) + require.NoError(t, transport.PublishFacts(ctx, key, authorFacts("bob"))) + + // The follower reads past the refused entry rather than stalling on it, so the + // fact appended after it still arrives. + entries := drainFactEntries(t, sub, 2) + require.Equal(t, []string{"alice", "bob"}, factAuthors(entries)) + + decodeErrors, found := telemetry.CollectInt64Sum(reader, + "gitopsreverser_attribution_fact_stream_decode_errors_total", + map[string]string{"transport": string(transport.TransportKind())}) + require.True(t, found, "a refused entry must be counted; it has no other symptom") + require.Equal(t, int64(1), decodeErrors) + }) + } + }) + } +} + func TestFactStreamConformance_EmptyBatchIsNotAppended(t *testing.T) { runFactStreamConformance(t, defaultFactStreamParams(), func(t *testing.T, transport FactTransport) { ctx := t.Context() @@ -465,3 +537,42 @@ func drainUntilFactStreamGap(t *testing.T, sub FactSubscription) (FactStreamGap, } } } + +// TestFactStreamConformance_ExactlyFullReadDoesNotReportAPhantomGap pins the trim-gap precondition +// against a FALSE positive, which is the other half of the test above. +// +// "Behind" is inferred from a read filling its entry budget, because a read that fills it exactly is +// indistinguishable from one that left more waiting. That inference is fine; leaving it set +// afterwards is not. A follower that fills the budget exactly and then finds nothing more is caught +// up — and if the mark survived, ordinary retention ageing out the entries it had ALREADY read +// would surface as a data-loss report and a scary log line for a follower that never lost anything. +func TestFactStreamConformance_ExactlyFullReadDoesNotReportAPhantomGap(t *testing.T) { + params := defaultFactStreamParams() + params.TTL = factStreamTestTTL + // One entry per read, and exactly one entry to read: the budget is filled precisely. + params.ReadCount = 1 + runFactStreamConformance(t, params, func(t *testing.T, transport FactTransport) { + ctx := t.Context() + key := factStreamTestKey("", "configmaps") + + require.NoError(t, transport.PublishFacts(ctx, key, authorFacts("only"))) + sub := transport.FollowFacts([]FactStreamKey{key}, factStreamTestHorizon) + require.Equal(t, []string{"only"}, factAuthors(drainFactEntries(t, sub, 1))) + + // It asks again and is given nothing: it has read everything there is. + empty, err := sub.Next(ctx) + require.NoError(t, err) + require.Empty(t, empty.Entries) + require.Empty(t, empty.Gaps) + + // Retention now ages out the entry it already read, and a new one arrives behind it. That is + // ordinary retention, not loss: this follower missed nothing. + time.Sleep(factStreamTestAgeWait) + require.NoError(t, transport.PublishFacts(ctx, key, authorFacts("next"))) + + delivery, err := sub.Next(ctx) + require.NoError(t, err) + require.Empty(t, delivery.Gaps, + "a follower that read everything there was must never be reported as trimmed past") + }) +} diff --git a/internal/queue/fact_stream_memory.go b/internal/queue/fact_stream_memory.go index 9212bd7c..1814a66e 100644 --- a/internal/queue/fact_stream_memory.go +++ b/internal/queue/fact_stream_memory.go @@ -97,16 +97,42 @@ func (m *MemoryFactStream) PublishFacts(ctx context.Context, key FactStreamKey, } ring.append(raw, now) ring.trim(now.Add(-m.ttl), m.maxLen) + m.dropIdleRings(now) close(m.signal) m.signal = make(chan struct{}) return nil } +// dropIdleRings forgets every stream whose entries have all aged out. Trimming is amortized onto +// the publish path, so without this a type that stops being written to keeps its last window of +// entries for the life of the process: nothing appends to it, so nothing ever trims it again. A +// namespace-scoped type garbage-collected once when a namespace is torn down is exactly that shape, +// and the memory is billed against a number that only ever grows — every (route, type) pair the +// process has ever seen. +// +// It runs under the publish lock, over a map holding one entry per followed type, so it is a walk +// of a few dozen entries on a path that already holds the lock. Its Redis counterpart is the EXPIRE +// that rides along with every XADD. +func (m *MemoryFactStream) dropIdleRings(now time.Time) { + horizon := now.Add(-m.ttl) + for key, ring := range m.streams { + ring.trim(horizon, m.maxLen) + if len(ring.entries) == 0 { + delete(m.streams, key) + } + } +} + // FollowFacts starts following keys from horizon before now. func (m *MemoryFactStream) FollowFacts(keys []FactStreamKey, horizon time.Duration) FactSubscription { return &memFactSubscription{stream: m, follow: newFollowSet(keys, horizon)} } +// TransportKind names this transport for the metric labels. +func (m *MemoryFactStream) TransportKind() FactTransportKind { + return FactTransportMemory +} + // wakeup returns the channel closed by the next publish. func (m *MemoryFactStream) wakeup() <-chan struct{} { m.mu.Lock() @@ -219,7 +245,7 @@ func (sub *memFactSubscription) Next(ctx context.Context) (FactDelivery, error) // Take the wake-up channel BEFORE reading, so a publish that lands between the read and // the wait is not missed: it closes the channel this iteration already holds. signal := sub.stream.wakeup() - delivery := sub.collect() + delivery := sub.collect(ctx) if len(delivery.Entries) > 0 || len(delivery.Gaps) > 0 { return delivery, nil } @@ -241,7 +267,7 @@ func (sub *memFactSubscription) Next(ctx context.Context) (FactDelivery, error) } // collect reads every followed ring once, advancing the cursors it delivered. -func (sub *memFactSubscription) collect() FactDelivery { +func (sub *memFactSubscription) collect(ctx context.Context) FactDelivery { targets := sub.follow.targets() var delivery FactDelivery for _, target := range targets { @@ -250,11 +276,13 @@ func (sub *memFactSubscription) collect() FactDelivery { delivery.Gaps = append(delivery.Gaps, gap) } if len(entries) == 0 { + sub.follow.caughtUp(target.Key) continue } for _, entry := range entries { facts, err := decodeFactBatch(entry.raw) if err != nil { + recordFactStreamDecodeError(ctx, sub.stream.TransportKind(), target.Key, entry.id, err) continue } delivery.Entries = append(delivery.Entries, FactEntry{Key: target.Key, ID: entry.id, Facts: facts}) diff --git a/internal/queue/fact_stream_memory_test.go b/internal/queue/fact_stream_memory_test.go new file mode 100644 index 00000000..7752db4e --- /dev/null +++ b/internal/queue/fact_stream_memory_test.go @@ -0,0 +1,36 @@ +// SPDX-License-Identifier: Apache-2.0 + +package queue + +import ( + "testing" + "time" + + "github.com/stretchr/testify/require" + "k8s.io/apimachinery/pkg/runtime/schema" +) + +// TestMemoryFactStream_ForgetsAStreamNobodyWritesToAnyMore pins the leak that trimming-on-publish +// leaves behind. A type that stops being written to is never trimmed again — nothing appends to it, +// and the trim rides on the append — so without an explicit sweep its last window of entries is +// held for the life of the process, billed against a number that only ever grows: every (route, +// type) pair the process has ever seen. A namespace-scoped type garbage-collected once when a +// namespace is torn down is exactly that shape. +func TestMemoryFactStream_ForgetsAStreamNobodyWritesToAnyMore(t *testing.T) { + stream := NewMemoryFactStream(MemoryFactStreamConfig{TTL: 40 * time.Millisecond}) + quiet := FactStreamKeyFor("prod-eu-1", schema.GroupResource{Resource: "challenges"}) + busy := FactStreamKeyFor("prod-eu-1", schema.GroupResource{Resource: "configmaps"}) + + require.NoError(t, stream.PublishFacts(t.Context(), quiet, + []AuthorFact{{UID: "uid-1", Author: "alice", Verb: "deletecollection"}})) + require.Len(t, stream.streams, 1) + + // The quiet stream is never written to again; the busy one keeps going. + time.Sleep(60 * time.Millisecond) + require.NoError(t, stream.PublishFacts(t.Context(), busy, + []AuthorFact{{UID: "uid-2", Author: "bob", Verb: "update"}})) + + require.NotContains(t, stream.streams, quiet, + "a stream past its retention horizon must be forgotten, not held for the life of the process") + require.Contains(t, stream.streams, busy, "a live stream must survive the sweep") +} diff --git a/internal/queue/fact_stream_redis.go b/internal/queue/fact_stream_redis.go index 678ac788..e1454ad4 100644 --- a/internal/queue/fact_stream_redis.go +++ b/internal/queue/fact_stream_redis.go @@ -108,7 +108,19 @@ func (s *RedisFactStream) PublishFacts(ctx context.Context, key FactStreamKey, f args.MaxLen = s.maxLen args.Approx = true } - if err := s.client.XAdd(ctx, args).Err(); err != nil { + // EXPIRE rides along with every append, so a stream whose type stops being written to deletes + // ITSELF one TTL later. Without it the keyspace only ever grows: MINID trimming is amortized + // onto the publish path, so a stream that goes quiet is never trimmed again and keeps its last + // entries for the life of the Redis instance. A namespace-scoped type that is garbage-collected + // once and never written again — the shape a torn-down namespace produces — would otherwise + // leave an immortal key behind for every (route, type) pair that ever saw one write. + // + // The deadline is refreshed by every append, so a busy stream never expires, and it matches the + // retention horizon, so the key dies exactly when its newest entry would have aged out anyway. + pipe := s.client.Pipeline() + pipe.XAdd(ctx, args) + pipe.Expire(ctx, streamKey, s.ttl) + if _, err := pipe.Exec(ctx); err != nil { return fmt.Errorf("append facts to %q: %w", streamKey, err) } s.trimIfDue(ctx, streamKey) @@ -120,6 +132,11 @@ func (s *RedisFactStream) FollowFacts(keys []FactStreamKey, horizon time.Duratio return &redisFactSubscription{stream: s, follow: newFollowSet(keys, horizon)} } +// TransportKind names this transport for the metric labels. +func (s *RedisFactStream) TransportKind() FactTransportKind { + return FactTransportRedis +} + // streamKey renders one stream's Redis key, e.g. // "gitops-reverser:author:v2:audit:route:prod-eu-1:apps/deployments". The route sits directly // after the domain so one route's streams share a single glob prefix. @@ -192,6 +209,12 @@ func (sub *redisFactSubscription) Next(ctx context.Context) (FactDelivery, error Block: sub.stream.block, }).Result() if errors.Is(err, redis.Nil) { + // The whole block period elapsed with nothing on any followed stream, so every one of them + // is caught up — including any the last read left marked behind for having exactly filled + // its entry budget. + for _, target := range targets { + sub.follow.caughtUp(target.Key) + } return FactDelivery{Gaps: gaps}, nil } if err != nil { @@ -200,16 +223,32 @@ func (sub *redisFactSubscription) Next(ctx context.Context) (FactDelivery, error sub.follow.forgetGaps(gaps) return FactDelivery{}, fmt.Errorf("read fact streams: %w", err) } - return FactDelivery{Entries: sub.collect(res, byStreamKey), Gaps: gaps}, nil + return FactDelivery{Entries: sub.collect(ctx, res, byStreamKey), Gaps: gaps}, nil } // collect turns one XREAD result into entries and advances the cursors it delivered. An entry // whose payload does not decode is skipped rather than retried: it can never decode, and stalling -// the follower on it would cost every later fact on that stream. +// the follower on it would cost every later fact on that stream. It is counted and logged, because +// skipping it loses its facts and leaves no other trace. func (sub *redisFactSubscription) collect( + ctx context.Context, res []redis.XStream, byStreamKey map[string]FactStreamKey, ) []FactEntry { + // XREAD returns only the streams that had something, so every followed stream missing from the + // result was asked and gave nothing: it is caught up, whatever the last read left marked. + delivered := make(map[FactStreamKey]struct{}, len(res)) + for i := range res { + if key, ok := byStreamKey[res[i].Stream]; ok && len(res[i].Messages) > 0 { + delivered[key] = struct{}{} + } + } + for _, key := range byStreamKey { + if _, ok := delivered[key]; !ok { + sub.follow.caughtUp(key) + } + } + var entries []FactEntry for i := range res { key, ok := byStreamKey[res[i].Stream] @@ -223,6 +262,7 @@ func (sub *redisFactSubscription) collect( for j := range messages { facts, err := factsFromMessage(messages[j]) if err != nil { + recordFactStreamDecodeError(ctx, sub.stream.TransportKind(), key, messages[j].ID, err) continue } entries = append(entries, FactEntry{Key: key, ID: messages[j].ID, Facts: facts}) diff --git a/internal/queue/fact_stream_redis_test.go b/internal/queue/fact_stream_redis_test.go index 32b3f5d3..4960bb6c 100644 --- a/internal/queue/fact_stream_redis_test.go +++ b/internal/queue/fact_stream_redis_test.go @@ -49,3 +49,34 @@ func TestRedisFactStream_TrimIsRateLimited(t *testing.T) { require.NoError(t, err) require.Len(t, entries, 2) } + +// TestRedisFactStream_StreamExpiresWhenNobodyWritesToItAnyMore pins the counterpart to the +// amortized trim above, and the leak it would otherwise leave. MINID trimming rides on the publish +// path, so a stream whose type stops being written to is never trimmed again and its key would +// live in Redis for the life of the instance — one immortal key per (route, type) pair that ever +// saw a single write. A namespace-scoped type garbage-collected once when a namespace is torn down +// is exactly that shape, and there is nothing left to clean it up. +// +// EXPIRE rides along with every XADD instead, so the key dies one retention horizon after its last +// append, and a busy stream keeps refreshing the deadline. +func TestRedisFactStream_StreamExpiresWhenNobodyWritesToItAnyMore(t *testing.T) { + store, mr := newTestRedisStoreWithRedis(t) + stream := store.FactStream(RedisFactStreamConfig{TTL: 10 * time.Minute}) + key := factStreamTestKey("acme.cert-manager.io", "challenges") + ctx := t.Context() + + require.NoError(t, stream.PublishFacts(ctx, key, authorFacts("alice"))) + streamKey := stream.streamKey(key) + require.Equal(t, 10*time.Minute, mr.TTL(streamKey), + "every append must set the retention deadline, or an idle stream never goes away") + + // A later append refreshes it rather than letting it run down, so a busy stream never expires. + mr.FastForward(9 * time.Minute) + require.NoError(t, stream.PublishFacts(ctx, key, authorFacts("bob"))) + require.Equal(t, 10*time.Minute, mr.TTL(streamKey), "a live stream's deadline must be refreshed") + + // Nothing writes to it again, so it goes. + mr.FastForward(11 * time.Minute) + require.False(t, mr.Exists(streamKey), + "a stream past its retention horizon must not outlive the facts it carried") +} diff --git a/internal/queue/key_prefix.go b/internal/queue/key_prefix.go index d2e5bd11..14a03eb5 100644 --- a/internal/queue/key_prefix.go +++ b/internal/queue/key_prefix.go @@ -12,14 +12,63 @@ import ( // dwarfs the key it namespaces. const maxKeyPrefixLength = 128 +// DefaultKeyPrefix is the root namespace every Redis key (cursors, fact streams, and command +// author records alike) carries when --redis-key-prefix is not set. It is also the value every +// release before the flag existed used, so the default is a no-op upgrade. +const DefaultKeyPrefix = "gitops-reverser" + +// routeKeyInfix carries the AUDIT ROUTE dimension so a fact from cluster A never joins a watch +// event from cluster B — the rv-only hatch especially, since RV is not globally unique. The route +// is what the audit events arrived under, NOT the ClusterProvider's name: an API server has one +// webhook backend and posts under one route, so several providers naming one cluster all declare +// that route and share its facts (ClusterProvider.AuditRoute()). It is spelled "route" rather than +// "auditRoute" because the key already says audit one segment earlier, and it sits directly after +// the domain suffix so one route's streams share a single glob prefix. +const routeKeyInfix = "route:" + +// groupResourceKey renders a GroupResource as an API-path-style segment: "configmaps" for the core +// group, "apps/deployments" otherwise. Publish side, follow side, and the index share it so the +// name never drifts. "/" never appears in a group or resource name, so the form stays unambiguously +// splittable — unlike schema.GroupResource.String()'s reversed dot form ("deployments.apps"), whose +// dot also collides with dotted group names. +func groupResourceKey(group, resource string) string { + if group == "" { + return resource + } + return group + "/" + resource +} + +// escapeKeyField neutralizes the ":" delimiter and the "%" escape character within a single key +// field. Group/resource and a UUID never contain either, so this is defensive for the uid and route +// fields against a stray delimiter; everything else passes through unchanged for readability. Keys +// are only ever matched exactly, never parsed back, so escaping is one-way. +func escapeKeyField(s string) string { + if !strings.ContainsAny(s, "%:") { + return s + } + var b strings.Builder + b.Grow(len(s)) + for i := range len(s) { + switch s[i] { + case '%': + b.WriteString("%25") + case ':': + b.WriteString("%3A") + default: + b.WriteByte(s[i]) + } + } + return b.String() +} + // ValidateKeyPrefix checks a --redis-key-prefix value and returns its normalized form. // // Two independent constraints shape the allowed character set: // -// - The attribution telemetry gauge SCANs ":author:v1:audit:*". Redis glob -// metacharacters (*, ?, [, ], \) in the prefix would silently make that pattern match -// the wrong keyspace, so they are rejected rather than escaped — a prefix is an -// operator-chosen identifier, not user data. +// - A prefix names a keyspace an operator inspects and, on a bad day, deletes by glob. Redis +// glob metacharacters (*, ?, [, ], \) in it would make ":*" match more than this +// install's keys, so they are rejected rather than escaped — a prefix is an operator-chosen +// identifier, not user data. // - Key fields (uid, resourceVersion, namespace) are ':'-delimited and %-escaped by // escapeKeyField. '%' in the prefix would make an escaped key ambiguous with an // unescaped one, so it is rejected too. diff --git a/internal/queue/key_prefix_test.go b/internal/queue/key_prefix_test.go index e4de21c4..6a9ae059 100644 --- a/internal/queue/key_prefix_test.go +++ b/internal/queue/key_prefix_test.go @@ -86,13 +86,9 @@ func TestRedisStore_KeyPrefixReachesEveryKeyFamily(t *testing.T) { "cell-a:tenant-7:watch:v1:target:gtuid-3:apps/deployments:namespace:team-a:last-rv", store.watchCursorKey("gtuid-3", gvr, "team-a")) - idx := store.AttributionIndex(0) - require.Equal(t, "cell-a:tenant-7:author:v1:audit:route:default:apps/deployments:object:uid-1:101", - idx.factKeyExact("default", "apps/deployments", "uid-1", "101")) - require.Equal(t, "cell-a:tenant-7:author:v1:audit:route:default:apps/deployments:object:uid-1:last", - idx.factKeyLast("default", "apps/deployments", "uid-1")) - require.Equal(t, "cell-a:tenant-7:author:v1:audit:route:default:apps/deployments:rv:101", - idx.factKeyRV("default", "apps/deployments", "101")) + stream := store.FactStream(RedisFactStreamConfig{}) + require.Equal(t, "cell-a:tenant-7:author:v2:audit:route:default:apps/deployments", + stream.streamKey(FactStreamKeyFor("default", gvr.GroupResource()))) require.Equal(t, "cell-a:tenant-7:author:v1:command:cr-uid", store.CommandAuthorStore().key("cr-uid")) } @@ -119,8 +115,9 @@ func TestRedisStore_ZeroValueStoreStillWritesPrefixedKeys(t *testing.T) { require.Equal(t, "gitops-reverser:watch:v1:target:gtuid-3:configmaps:cluster:last-rv", store.watchCursorKey("gtuid-3", coreConfigmapsGVR(), "")) require.Equal(t, "gitops-reverser:author:v1:command:cr-uid", store.CommandAuthorStore().key("cr-uid")) - require.Equal(t, "gitops-reverser:author:v1:audit:route:default:", - store.AttributionIndex(0).routeFactPrefix("default")) + require.Equal(t, "gitops-reverser:author:v2:audit:route:default:configmaps", + store.FactStream(RedisFactStreamConfig{}).streamKey( + FactStreamKeyFor("default", coreConfigmapsGVR().GroupResource()))) } // Two reversers sharing one Redis/Valkey and one logical database must not read each @@ -153,14 +150,25 @@ func TestRedisStore_DistinctPrefixesIsolateCursors(t *testing.T) { require.Equal(t, "111", rv, "tenant-b's write must not clobber tenant-a's cursor") } -// The attribution telemetry gauge SCANs ":author:v1:audit:*" and the per-provider purge -// SCANs ":author:v1:audit:route::*". A prefix that contained a glob metacharacter -// would make either count/delete the wrong keyspace; validation rejects those, so the pattern is -// always a literal prefix plus one trailing star. -func TestAttributionIndex_ScanPatternIsPrefixed(t *testing.T) { +// A route's streams share one glob prefix, so an operator can inspect or purge exactly one audit +// route's facts with ":author:v2:audit:route::*". Validation rejects a prefix carrying +// a glob metacharacter, so that pattern is always a literal prefix plus one trailing star. +func TestRedisFactStream_RouteStreamsShareOneGlobPrefix(t *testing.T) { t.Parallel() - store := newPrefixedRedisStore(t, "tenant-a") - idx := store.AttributionIndex(0) - require.Equal(t, "tenant-a:author:v1:audit:route:prod-eu-1:", idx.routeFactPrefix("prod-eu-1")) + stream := newPrefixedRedisStore(t, "tenant-a").FactStream(RedisFactStreamConfig{}) + const routePrefix = "tenant-a:author:v2:audit:route:prod-eu-1:" + + for _, gr := range []schema.GroupResource{ + {Resource: "configmaps"}, + {Group: "apps", Resource: "deployments"}, + } { + require.True(t, + strings.HasPrefix(stream.streamKey(FactStreamKeyFor("prod-eu-1", gr)), routePrefix), + "every stream on a route must sit under that route's prefix") + } + // Another route's stream must NOT: the prefix is what makes a per-route purge safe. + require.False(t, strings.HasPrefix( + stream.streamKey(FactStreamKeyFor("prod-us-1", schema.GroupResource{Resource: "configmaps"})), + routePrefix)) } diff --git a/internal/queue/redis_store.go b/internal/queue/redis_store.go index 6a7500d8..0c1cb1d6 100644 --- a/internal/queue/redis_store.go +++ b/internal/queue/redis_store.go @@ -41,11 +41,15 @@ type RedisStoreConfig struct { KeyPrefix string } -// RedisStore is the required Redis/Valkey-backed store. It owns the connection and -// persists each GitTarget watch shard's resume cursor (state continuity / work -// re-pickup), and the readiness gate pings it. It is a hard dependency in every mode -// and knows nothing about attribution: author attribution is an optional layer built -// on the same connection via AttributionIndex. +// RedisStore is the optional Redis/Valkey-backed store. It owns the connection and persists each +// GitTarget watch shard's resume cursor (state continuity / work re-pickup), and the readiness gate +// pings it. It is NOT a hard dependency: --redis-addr may be empty, in which case watches +// cold-replay on restart instead of resuming. What does require it is the admission webhook's +// command-author capture, and attribution when it runs over the Redis fact-stream transport; +// attribution over the in-memory transport needs no Redis at all. +// +// It knows nothing about attribution facts. Those live in their own streams, built on the same +// connection by RedisFactStream. type RedisStore struct { client *redis.Client keyPrefix string @@ -76,16 +80,6 @@ func (s *RedisStore) KeyPrefix() string { return s.keyPrefix } -// AttributionIndex builds the optional author-attribution fact index on this store's -// connection. Call it only when author attribution is enabled — the store itself, -// and the resume cursors it holds, never depend on it. -func (s *RedisStore) AttributionIndex(factTTL time.Duration) *AttributionIndex { - if factTTL <= 0 { - factTTL = DefaultAttributionFactTTL - } - return &AttributionIndex{client: s.client, factTTL: factTTL, keyPrefix: s.keyPrefix} -} - // CommandAuthorStore builds the command-authorship store on this connection. Wire it // when the validate-operator-types webhook is enabled; it does not depend on attribution. The // record lives in the same top-level author domain as audit facts but in the separate diff --git a/internal/queue/redis_store_test.go b/internal/queue/redis_store_test.go new file mode 100644 index 00000000..5a46abd8 --- /dev/null +++ b/internal/queue/redis_store_test.go @@ -0,0 +1,283 @@ +// SPDX-License-Identifier: Apache-2.0 + +package queue + +import ( + "context" + "fmt" + "testing" + "time" + + "github.com/alicebob/miniredis/v2" + "github.com/stretchr/testify/require" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + auditv1 "k8s.io/apiserver/pkg/apis/audit/v1" +) + +func newTestRedisStore(t *testing.T) *RedisStore { + t.Helper() + store, _ := newTestRedisStoreWithRedis(t) + return store +} + +func newTestRedisStoreWithRedis(t *testing.T) (*RedisStore, *miniredis.Miniredis) { + t.Helper() + mr := miniredis.RunT(t) + store, err := NewRedisStore(RedisStoreConfig{Addr: mr.Addr()}) + require.NoError(t, err) + return store, mr +} + +// coreConfigmapsGVR is a core-group type, whose key form drops the group segment. +func coreConfigmapsGVR() schema.GroupVersionResource { + return schema.GroupVersionResource{Group: "", Version: "v1", Resource: "configmaps"} +} + +func TestRedisStore_Ping(t *testing.T) { + store := newTestRedisStore(t) + require.NoError(t, store.Ping(context.Background())) +} + +func TestRedisStore_WatchCursorRoundTrip(t *testing.T) { + store, mr := newTestRedisStoreWithRedis(t) + ctx := context.Background() + gvr := appsDeploymentGVR() + + _, ok := store.LookupWatchCursor(ctx, "uid-1", gvr, "apps") + require.False(t, ok) + + require.NoError(t, store.RecordWatchCursor(ctx, "uid-1", gvr, "apps", "42")) + got, ok := store.LookupWatchCursor(ctx, "uid-1", gvr, "apps") + require.True(t, ok) + require.Equal(t, "42", got) + + // The cursor carries watchCursorTTL and is never deleted explicitly; it expires + // once a watch has been gone longer than the TTL. + require.Equal(t, watchCursorTTL, mr.TTL(store.watchCursorKey("uid-1", gvr, "apps"))) + mr.FastForward(watchCursorTTL + time.Second) + _, ok = store.LookupWatchCursor(ctx, "uid-1", gvr, "apps") + require.False(t, ok) +} + +func TestRedisStore_WatchCursorIsolatedByGitTargetUID(t *testing.T) { + store := newTestRedisStore(t) + ctx := context.Background() + gvr := appsDeploymentGVR() + + require.NoError(t, store.RecordWatchCursor(ctx, "uid-old", gvr, "apps", "42")) + + // A GitTarget recreated under the same namespace/name but a new UID must not + // inherit its predecessor's cursor. + _, ok := store.LookupWatchCursor(ctx, "uid-new", gvr, "apps") + require.False(t, ok) + + got, ok := store.LookupWatchCursor(ctx, "uid-old", gvr, "apps") + require.True(t, ok) + require.Equal(t, "42", got) +} + +func TestRedisStore_WatchCursorIgnoresEmptyResourceVersion(t *testing.T) { + store := newTestRedisStore(t) + ctx := context.Background() + + require.NoError(t, store.RecordWatchCursor(ctx, "uid-1", appsDeploymentGVR(), "apps", "")) + _, ok := store.LookupWatchCursor(ctx, "uid-1", appsDeploymentGVR(), "apps") + require.False(t, ok) +} + +func TestRedisStore_WatchCursorKeyReadableFormat(t *testing.T) { + store := newTestRedisStore(t) + gvr := schema.GroupVersionResource{Group: "apps", Version: "v1", Resource: "deployments"} + + require.Equal(t, "gitops-reverser:watch:v1:target:gtuid-3:apps/deployments:namespace:team-a:last-rv", + store.watchCursorKey("gtuid-3", gvr, "team-a")) + + // A cluster-wide watch (empty namespace) uses the cluster scope segment, and the GVR + // version is dropped. + require.Equal(t, "gitops-reverser:watch:v1:target:gtuid-3:configmaps:cluster:last-rv", + store.watchCursorKey("gtuid-3", coreConfigmapsGVR(), "")) +} + +func TestNewRedisStore_RequiresAddr(t *testing.T) { + _, err := NewRedisStore(RedisStoreConfig{}) + require.Error(t, err) +} + +func TestEscapeKeyField(t *testing.T) { + cases := []struct{ in, want string }{ + {"web", "web"}, + {"101", "101"}, + {"rbac.authorization.k8s.io", "rbac.authorization.k8s.io"}, + {"system:node-proxier", "system%3Anode-proxier"}, + {"a%b", "a%25b"}, + {"%3A", "%253A"}, // a literal "%3A" must stay distinct from an escaped colon + {"", ""}, + } + for _, c := range cases { + require.Equal(t, c.want, escapeKeyField(c.in), "escapeKeyField(%q)", c.in) + } +} + +func TestGroupResourceKey(t *testing.T) { + require.Equal(t, "configmaps", groupResourceKey("", "configmaps")) + require.Equal(t, "apps/deployments", groupResourceKey("apps", "deployments")) + require.Equal(t, "rbac.authorization.k8s.io/roles", groupResourceKey("rbac.authorization.k8s.io", "roles")) +} + +// rawObject wraps a JSON body the way the audit pipeline delivers request/response bodies. + +func rawObject(body string) *runtime.Unknown { + return &runtime.Unknown{Raw: []byte(body)} +} + +// deploymentBody renders a minimal Deployment whose metadata.resourceVersion is rv. +func deploymentBody(rv string) string { + return fmt.Sprintf(`{"apiVersion":"apps/v1","kind":"Deployment",`+ + `"metadata":{"name":"web","namespace":"team-a","uid":"uid-1","resourceVersion":%q}}`, rv) +} + +// TestResourceVersionFromEvent_Precedence pins the RV precedence the join depends on. The RV is +// half of the fact key, so reading the wrong one files the fact under an object version that will +// never be looked up — the write silently ships the committer instead of its real author. Only the +// POST-write RV identifies the version a mutation produced: that lives in responseObject. +// requestObject carries the PRE-write RV and must never be consulted, and objectRef.resourceVersion +// is usually the empty precondition RV on writes, so it is a last resort only. +func TestResourceVersionFromEvent_Precedence(t *testing.T) { + cases := []struct { + name string + mutate func(*auditv1.Event) + wantRV string + wantWhy string + }{ + { + name: "response object wins over a different objectRef RV", + mutate: func(e *auditv1.Event) { + e.ResponseObject = rawObject(deploymentBody("202")) + e.ObjectRef.ResourceVersion = "101" + }, + wantRV: "202", + wantWhy: "the post-write RV in the response body is authoritative", + }, + { + name: "request object is ignored even when the response object has none", + mutate: func(e *auditv1.Event) { + e.RequestObject = rawObject(deploymentBody("101")) + e.ResponseObject = nil + e.ObjectRef.ResourceVersion = "" + }, + wantRV: "", + wantWhy: "requestObject holds the pre-write RV and is never a source", + }, + { + name: "request object never outranks the response object", + mutate: func(e *auditv1.Event) { + e.RequestObject = rawObject(deploymentBody("101")) + e.ResponseObject = rawObject(deploymentBody("202")) + }, + wantRV: "202", + wantWhy: "responseObject is consulted first and short-circuits", + }, + { + name: "objectRef is the fallback when the response object is absent", + mutate: func(e *auditv1.Event) { + e.ResponseObject = nil + e.ObjectRef.ResourceVersion = "101" + }, + wantRV: "101", + wantWhy: "objectRef is the last resort, not the first choice", + }, + { + name: "objectRef is the fallback when the response body carries no RV", + mutate: func(e *auditv1.Event) { + e.ResponseObject = rawObject(`{"metadata":{"name":"web"}}`) + e.ObjectRef.ResourceVersion = "101" + }, + wantRV: "101", + wantWhy: "a shallow body yields nothing, so the fallback still applies", + }, + { + name: "objectRef is the fallback when the response body is malformed", + mutate: func(e *auditv1.Event) { + e.ResponseObject = rawObject(`{"metadata":`) + e.ObjectRef.ResourceVersion = "101" + }, + wantRV: "101", + wantWhy: "an unparseable body must not poison the fallback", + }, + { + name: "empty precondition RV on objectRef yields nothing", + mutate: func(e *auditv1.Event) { + e.ResponseObject = nil + e.ObjectRef.ResourceVersion = "" + }, + wantRV: "", + wantWhy: "writes usually leave objectRef.resourceVersion empty", + }, + { + name: "nil objectRef and nil response object yield nothing", + mutate: func(e *auditv1.Event) { + e.ResponseObject = nil + e.ObjectRef = nil + }, + wantRV: "", + wantWhy: "collection verbs and deletes legitimately have no RV", + }, + { + name: "nil objectRef does not fall through to the request object", + mutate: func(e *auditv1.Event) { + e.ResponseObject = nil + e.ObjectRef = nil + e.RequestObject = rawObject(deploymentBody("101")) + }, + wantRV: "", + wantWhy: "requestObject stays ignored on every path", + }, + } + + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + event := mutationEvent("update", "uid-1", "202", "alice") + c.mutate(&event) + require.Equal(t, c.wantRV, resourceVersionFromEvent(event), c.wantWhy) + }) + } +} + +// TestRVFromRawObject_Cases covers the body shapes the audit stream actually delivers. Every +// non-answer must be "" rather than a partial or panicking read: a truncated or bodyless audit +// event has to degrade into "no RV recorded", not into a bogus RV that keys a fact nobody finds. +func TestRVFromRawObject_Cases(t *testing.T) { + cases := []struct { + name string + obj *runtime.Unknown + want string + }{ + {name: "nil object", obj: nil, want: ""}, + {name: "nil raw bytes", obj: &runtime.Unknown{}, want: ""}, + {name: "zero-length raw bytes", obj: &runtime.Unknown{Raw: []byte{}}, want: ""}, + {name: "malformed json", obj: rawObject(`{"metadata":{"resourceVersion":`), want: ""}, + {name: "non-object json", obj: rawObject(`"a string"`), want: ""}, + {name: "empty json object", obj: rawObject(`{}`), want: ""}, + {name: "object without metadata", obj: rawObject(`{"kind":"Deployment"}`), want: ""}, + {name: "metadata without resourceVersion", obj: rawObject(`{"metadata":{"name":"web"}}`), want: ""}, + {name: "explicitly empty resourceVersion", obj: rawObject(`{"metadata":{"resourceVersion":""}}`), want: ""}, + {name: "well-formed body", obj: rawObject(deploymentBody("101")), want: "101"}, + { + name: "resourceVersion alongside unknown fields", + obj: rawObject(`{"spec":{"replicas":3},"metadata":{"name":"web","resourceVersion":"99"}}`), + want: "99", + }, + } + + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + require.Equal(t, c.want, rvFromRawObject(c.obj)) + }) + } +} + +// TestAttributionIndex_CrossClusterIsolation is the multi-cluster centerpiece: two clusters +// record the SAME object identity (uid, rv) with different authors, and each cluster's read joins +// ONLY its own fact — never the other's. A third cluster that recorded nothing misses (ships +// committer) instead of borrowing a neighbor's author. diff --git a/internal/telemetry/exporter.go b/internal/telemetry/exporter.go index 6e34950e..ce1c90e5 100644 --- a/internal/telemetry/exporter.go +++ b/internal/telemetry/exporter.go @@ -98,29 +98,59 @@ var ( // AuditEventListDurationSeconds records how long the webhook takes to answer an // EventList request, labelled by outcome. AuditEventListDurationSeconds metric.Float64Histogram - // AttributionResolutionsTotal counts watch-event attribution resolver outcomes, - // labelled by {result, group, version, resource}. + // AttributionResolutionsTotal counts watch-event attribution resolver outcomes, labelled by + // {tier, actor_kind, group, version, resource}. tier names WHICH evidence answered + // (exact/latest/resource_version/name/collection_uid/collection_scope/absent) and actor_kind + // names WHO it named (user/serviceaccount/none) — two orthogonal questions, so they are two + // labels. Match coverage is tier!="absent"; anything narrower reads the collection and name + // tiers as misses. AttributionResolutionsTotal metric.Int64Counter - // AttributionFactEventsTotal counts attribution fact lifecycle events in Redis, - // labelled by bounded op (written/matched/expired_unmatched/late). - AttributionFactEventsTotal metric.Int64Counter - // AttributionResolutionWaitSeconds records resolver wait time by final result. + // AttributionFactsTotal counts attribution fact lifecycle events, labelled by bounded op: + // "written" is one fact appended to the fact log, "matched" is one joined by a watch event. + // Together they say how much of what is published is ever used. They are NOT subtractable: + // written counts every type, matched only the streams this process follows. + AttributionFactsTotal metric.Int64Counter + // AttributionResolutionWaitSeconds records resolver wait time by {tier, event_kind, group, + // version, resource}. event_kind is write or removal, and the split is load-bearing: a removal + // holds a fallback and keeps waiting where a write does not, so the removal wait is the number + // --author-attribution-grace is tuned from. AttributionResolutionWaitSeconds metric.Float64Histogram - // AttributionFactIndexSize gauges attribution fact keys currently held in Redis. - AttributionFactIndexSize metric.Int64Gauge + // AttributionFactIndexEntries gauges the entries the in-memory fact index currently holds across + // every scope and match structure. Read against the eviction counter it says whether the caps + // are binding. + AttributionFactIndexEntries metric.Int64Gauge // AttributionFactIndexEvictionsTotal counts facts dropped from the in-memory fact index because // it was full, labelled by bounded reason (per_type/total). An attribution lost to a full index // has to look different from one that was never published, or a burst is silently absorbed. AttributionFactIndexEvictionsTotal metric.Int64Counter - // AttributionCollectionDegradedTotal counts collection facts published without the uid set the - // precise join would have used, labelled by bounded reason (uid_cap/no_uids). The scope fallback - // is already correct, so this says how often the precise path was available — not that anything - // broke. - AttributionCollectionDegradedTotal metric.Int64Counter + // AttributionCollectionWithoutUIDSetTotal counts collection facts published without the uid set + // the precise join would have used, labelled by bounded reason (uid_cap/no_uids). The scope + // fallback is already correct, so this says how often the precise path was available — not that + // anything broke. + AttributionCollectionWithoutUIDSetTotal metric.Int64Counter // AttributionFactStreamGapsTotal counts occasions a fact stream was trimmed past this process's // follower, labelled by stream. Every gap is facts lost for good, and it is the one loss a log // transport can see at all. AttributionFactStreamGapsTotal metric.Int64Counter + // AttributionFactStreamDecodeErrorsTotal counts fact-stream entries the follower could not + // decode, labelled by transport. Such an entry is skipped and its position passed, so the facts + // it carried are lost — and unlike a trim gap the loss leaves no other trace, which is why this + // is the loss path that most needed a counter. + AttributionFactStreamDecodeErrorsTotal metric.Int64Counter + // AttributionFactFollowerErrorsTotal counts fact-follower read failures, labelled by transport. + // The follower retries with a backoff rather than returning, so the errors are otherwise only a + // log line. + AttributionFactFollowerErrorsTotal metric.Int64Counter + // AttributionFactFollowerLastSuccessTimestampSeconds gauges the Unix time of the follower's last + // successful read, idle rounds included. It matters more than the error counter: only it + // separates "erroring occasionally while making progress" from "has read nothing in ten + // minutes", and only the second is an outage. Read it as time() - . + AttributionFactFollowerLastSuccessTimestampSeconds metric.Int64Gauge + // AttributionTransportInfo is an info gauge, always 1, labelled by the fact transport in force + // (redis/memory). It is interpretive metadata rather than a signal: a burst of unresolved + // commits after a restart is EXPECTED under the in-process transport, which drops every fact + // with the process, and a bug under Redis. + AttributionTransportInfo metric.Int64Gauge // APICatalogResources gauges the count of served top-level resources in the catalog, // split by the default-watch-policy allowed/excluded state. @@ -236,10 +266,18 @@ func registerCounters() error { {"gitopsreverser_audit_eventlists_total", &AuditEventListsTotal}, {"gitopsreverser_audit_eventlist_events_total", &AuditEventListEventsTotal}, {"gitopsreverser_attribution_resolutions_total", &AttributionResolutionsTotal}, - {"gitopsreverser_attribution_fact_events_total", &AttributionFactEventsTotal}, + {"gitopsreverser_attribution_facts_total", &AttributionFactsTotal}, {"gitopsreverser_attribution_fact_index_evictions_total", &AttributionFactIndexEvictionsTotal}, {"gitopsreverser_attribution_fact_stream_gaps_total", &AttributionFactStreamGapsTotal}, - {"gitopsreverser_attribution_collection_degraded_total", &AttributionCollectionDegradedTotal}, + { + "gitopsreverser_attribution_collection_without_uidset_total", + &AttributionCollectionWithoutUIDSetTotal, + }, + { + "gitopsreverser_attribution_fact_stream_decode_errors_total", + &AttributionFactStreamDecodeErrorsTotal, + }, + {"gitopsreverser_attribution_fact_follower_errors_total", &AttributionFactFollowerErrorsTotal}, {"gitopsreverser_api_catalog_refresh_total", &APICatalogRefreshTotal}, {"gitopsreverser_secret_encryption_attempts_total", &SecretEncryptionAttemptsTotal}, {"gitopsreverser_secret_encryption_success_total", &SecretEncryptionSuccessTotal}, @@ -301,7 +339,12 @@ func registerGauges() error { {"gitopsreverser_api_catalog_generation", &APICatalogGeneration}, {"gitopsreverser_watched_types", &WatchedTypes}, {"gitopsreverser_branch_worker_queue_depth", &BranchWorkerQueueDepth}, - {"gitopsreverser_attribution_fact_index_size", &AttributionFactIndexSize}, + {"gitopsreverser_attribution_fact_index_entries", &AttributionFactIndexEntries}, + { + "gitopsreverser_attribution_fact_follower_last_success_timestamp_seconds", + &AttributionFactFollowerLastSuccessTimestampSeconds, + }, + {"gitopsreverser_attribution_transport_info", &AttributionTransportInfo}, } for _, s := range gauges { v, err := otelMeter.Int64Gauge(s.name) diff --git a/internal/telemetry/exporter_test.go b/internal/telemetry/exporter_test.go index f2080a64..eb4cc90c 100644 --- a/internal/telemetry/exporter_test.go +++ b/internal/telemetry/exporter_test.go @@ -41,9 +41,14 @@ func TestInitOTLPExporter_Success(t *testing.T) { assert.NotNil(t, AuditEventListDurationSeconds) assert.NotNil(t, AuditEventsTotal) assert.NotNil(t, AttributionResolutionsTotal) - assert.NotNil(t, AttributionFactEventsTotal) + assert.NotNil(t, AttributionFactsTotal) assert.NotNil(t, AttributionResolutionWaitSeconds) - assert.NotNil(t, AttributionFactIndexSize) + assert.NotNil(t, AttributionFactIndexEntries) + assert.NotNil(t, AttributionCollectionWithoutUIDSetTotal) + assert.NotNil(t, AttributionFactStreamDecodeErrorsTotal) + assert.NotNil(t, AttributionFactFollowerErrorsTotal) + assert.NotNil(t, AttributionFactFollowerLastSuccessTimestampSeconds) + assert.NotNil(t, AttributionTransportInfo) assert.NotNil(t, APICatalogResources) assert.NotNil(t, APICatalogGroupVersions) assert.NotNil(t, APICatalogRefreshTotal) diff --git a/internal/watch/author_resolver.go b/internal/watch/author_resolver.go index 7b5cbaaf..dbebee62 100644 --- a/internal/watch/author_resolver.go +++ b/internal/watch/author_resolver.go @@ -24,25 +24,60 @@ import ( // enforceable: we wait briefly BEFORE shipping rather than rewrite afterwards. const DefaultAttributionGraceWindow = 3 * time.Second -// attributionPollInterval is how often the resolver re-checks the index while it -// waits out the grace window for a fact that has not arrived yet. -const attributionPollInterval = 150 * time.Millisecond - -// AttributionLookup is the read side of the optional audit attribution index. The -// Redis-backed queue.AttributionIndex satisfies it; nil means configured-author. +// AttributionLookup is the read side of the attribution fact index. The in-process +// queue.FactIndex satisfies it; nil means configured-author. +// +// It waits rather than polls. The index registers the waiter BEFORE it reads itself, so a fact +// delivered in the gap between the two wakes a waiter that is already listening — the race the old +// 150ms poll loop papered over by looking again. There is no Redis call anywhere on this path: the +// fast case is a map read and the waiting case is a channel receive. type AttributionLookup interface { - // LookupAuthorResolution resolves the strongest author fact for a watch event. - // exactCapable is true for ADDED/MODIFIED events (try only the immutable exact key - // and the rv-only hatch) and false for known RV-mismatch events such as DELETED - // (also consult the last-writer-wins /last pointer). - LookupAuthorResolution( - ctx context.Context, - auditRoute string, - gvr schema.GroupVersionResource, - uid k8stypes.UID, - rv string, - exactCapable bool, - ) queue.AuthorResolution + // Await resolves the strongest author fact for a watch event, waiting up to grace for one that + // has not been delivered yet. It returns an AttributionAbsent resolution when nothing matched in + // time; it never blocks longer than grace and never returns an error path. + Await(ctx context.Context, query queue.FactQuery, grace time.Duration) queue.AuthorResolution +} + +// AuthorQuery is one watch event's identity, as the author resolver reads it. +// +// It is a struct rather than the parameter list this used to be because the collection tier needs +// the object's namespace and labels: a body-less deletecollection is joined by scope — type, +// namespace, selector, window — and neither the namespace nor the labels could be expressed in the +// old six arguments, so that tier would have been unreachable from the resolver. +type AuthorQuery struct { + // AuditRoute partitions the facts. It is the route the API server posts audit under, NOT the + // ClusterProvider's name: several providers may name one cluster and all read its facts. + AuditRoute string + // GVR is the watched type. The version serves the metric labels only; the join is keyed on the + // group/resource, which is what a fact carries. + GVR schema.GroupVersionResource + UID k8stypes.UID + ResourceVersion string + // Namespace and Labels serve the collection tier only: they are how a removal finds the + // deletecollection whose scope covered it. + Namespace string + Labels map[string]string + // Name serves the name tier only, the floor for a fact carrying neither a uid nor a + // resourceVersion — an aggregated-API write, whose audit objectRef holds the name and nothing + // else. The watch event always carries it, so supplying it costs nothing when no fact needs it. + Name string + // ExactCapable is true for ADDED and MODIFIED, whose resourceVersion is the one the write + // produced. A removal's is not, so it consults the weaker tiers the exact-capable events skip. + ExactCapable bool +} + +// factQuery renders the query the way the index is keyed. +func (q AuthorQuery) factQuery() queue.FactQuery { + return queue.FactQuery{ + AuditRoute: q.AuditRoute, + GroupResource: q.GVR.GroupResource(), + UID: string(q.UID), + ResourceVersion: q.ResourceVersion, + Namespace: q.Namespace, + Labels: q.Labels, + Name: q.Name, + ExactCapable: q.ExactCapable, + } } // CursorStore persists the last processed resourceVersion for each (GitTarget UID, @@ -82,15 +117,8 @@ type AuthorResolver interface { // In production this method only ever returns the latter two: configured-author mode is // expressed by leaving Manager.AuthorResolver nil (attachAuthor returns early, leaving the // event's zero AttributionNotAttempted), never by constructing a resolver over a nil - // lookup. cmd/main.go:258 only builds one with a non-nil index. - ResolveAuthor( - ctx context.Context, - auditRoute string, - gvr schema.GroupVersionResource, - uid k8stypes.UID, - rv string, - exactCapable bool, - ) (git.UserInfo, git.AttributionOutcome) + // lookup. cmd/main.go only builds one with a non-nil index. + ResolveAuthor(ctx context.Context, query AuthorQuery) (git.UserInfo, git.AttributionOutcome) } // attributionUnresolvedWarnThreshold is how many consecutive unresolved events one audit route may @@ -159,42 +187,43 @@ func NewAuthorResolver( func (r *attributionResolver) ResolveAuthor( ctx context.Context, - auditRoute string, - gvr schema.GroupVersionResource, - uid k8stypes.UID, - rv string, - exactCapable bool, + query AuthorQuery, ) (git.UserInfo, git.AttributionOutcome) { start := time.Now() // A nil lookup is configured-author mode: attribution was never switched on, so nothing // was attempted and the committer legitimately authors the commit. Defensive only — // production expresses that mode with a nil Manager.AuthorResolver, so this branch is - // unreachable there (cmd/main.go:258 always passes a non-nil index). + // unreachable there (cmd/main.go always passes a non-nil index). if r.lookup == nil { - recordAttributionResolution(ctx, gvr, queue.AttributionAbsent, time.Since(start)) + recordAttributionResolution(ctx, query, queue.AttributionAbsent, queue.ActorKindNone, time.Since(start)) return git.UserInfo{}, git.AttributionNotAttempted } - deadline := time.Now().Add(r.grace) - for { - resolution := r.lookup.LookupAuthorResolution(ctx, auditRoute, gvr, uid, rv, exactCapable) - if resolution.Result != queue.AttributionAbsent { - ui, outcome, result := r.userInfoForResolution(resolution) - recordAttributionResolution(ctx, gvr, result, time.Since(start)) - r.health.observe(auditRoute, outcome == git.AttributionResolved) - return ui, outcome - } - if !time.Now().Before(deadline) || !sleepOrDone(ctx, attributionPollInterval) { - recordAttributionResolution(ctx, gvr, queue.AttributionAbsent, time.Since(start)) - r.warnIfRouteNeverResolves(auditRoute, gvr) - return git.UserInfo{}, git.AttributionUnresolved - } + // One call, not a loop. The whole wait — register the waiter, read the index, block on the + // waiter or the grace deadline — belongs to the index, which is the only thing that knows when a + // fact arrives. AttributionResolutionWaitSeconds still measures the same span it always did: + // entry to outcome on the watch shard's own goroutine. + resolution := r.lookup.Await(ctx, query.factQuery(), r.grace) + if resolution.Result != queue.AttributionAbsent { + ui, outcome, result := r.userInfoForResolution(resolution) + recordAttributionResolution(ctx, query, result, resolution.ActorKind(), time.Since(start)) + r.health.observe(query.AuditRoute, outcome == git.AttributionResolved) + return ui, outcome } + recordAttributionResolution(ctx, query, queue.AttributionAbsent, queue.ActorKindNone, time.Since(start)) + r.warnIfRouteNeverResolves(query.AuditRoute, query.GVR) + return git.UserInfo{}, git.AttributionUnresolved } // userInfoForResolution turns a matched fact into a commit author. The matched // actor — human or service account — is always named by its own username; a fact // that carries no author is UNRESOLVED, not not-attempted: attribution ran, found a // fact, and still could not name anyone. +// +// Both ends now make that branch unreachable rather than merely unlikely — the publish gate refuses +// an event whose user cannot be resolved, and the fact's own UnmarshalJSON refuses an entry that +// names nobody — so it survives for the zero-value paths and to keep the metric honest if a fact +// ever reaches here without an author: it is recorded with its tier and actor_kind="none", never as +// a named actor. func (r *attributionResolver) userInfoForResolution( resolution queue.AuthorResolution, ) (git.UserInfo, git.AttributionOutcome, queue.AttributionResult) { @@ -210,24 +239,52 @@ func (r *attributionResolver) userInfoForResolution( }, git.AttributionResolved, result } +// attributionEventKindWrite and attributionEventKindRemoval are the bounded event kinds on the wait +// histogram. They come from ExactCapable, which is the same split the wait design turns on: a +// removal holds a fallback and keeps waiting for evidence about the deletion, a write does not. An +// absent write and an absent removal used to be one series, and the removal wait is the number +// anyone tuning --author-attribution-grace actually needs. +const ( + attributionEventKindWrite = "write" + attributionEventKindRemoval = "removal" +) + +// recordAttributionResolution counts one resolution and times it. +// +// The two instruments carry DIFFERENT label sets on purpose. actor_kind is a property of the answer +// and belongs on the census; event_kind is a property of the question and only changes what the +// wait means. Putting both on both would multiply a histogram that already carries the type triple +// by six for no reading anyone would make. func recordAttributionResolution( ctx context.Context, - gvr schema.GroupVersionResource, - result queue.AttributionResult, + query AuthorQuery, + tier queue.AttributionResult, + actorKind queue.ActorKind, wait time.Duration, ) { - attrs := metric.WithAttributes( - attribute.String("result", string(result)), + gvr := query.GVR + typeAttrs := []attribute.KeyValue{ + attribute.String("tier", string(tier)), attribute.String("group", gvr.Group), attribute.String("version", gvr.Version), attribute.String("resource", gvr.Resource), - ) + } if telemetry.AttributionResolutionsTotal != nil { - telemetry.AttributionResolutionsTotal.Add(ctx, 1, attrs) + telemetry.AttributionResolutionsTotal.Add(ctx, 1, metric.WithAttributes( + append(typeAttrs, attribute.String("actor_kind", string(actorKind)))...)) } if telemetry.AttributionResolutionWaitSeconds != nil { - telemetry.AttributionResolutionWaitSeconds.Record(ctx, wait.Seconds(), attrs) + telemetry.AttributionResolutionWaitSeconds.Record(ctx, wait.Seconds(), metric.WithAttributes( + append(typeAttrs, attribute.String("event_kind", attributionEventKind(query)))...)) + } +} + +// attributionEventKind names whether the query was about a write or a removal. +func attributionEventKind(query AuthorQuery) string { + if query.ExactCapable { + return attributionEventKindWrite } + return attributionEventKindRemoval } // warnIfRouteNeverResolves says, once per audit route, that a route has produced a run of diff --git a/internal/watch/author_resolver_index_test.go b/internal/watch/author_resolver_index_test.go new file mode 100644 index 00000000..50b9742b --- /dev/null +++ b/internal/watch/author_resolver_index_test.go @@ -0,0 +1,208 @@ +// SPDX-License-Identifier: Apache-2.0 + +package watch + +import ( + "context" + "testing" + "time" + + "github.com/go-logr/logr" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "k8s.io/apimachinery/pkg/runtime/schema" + k8stypes "k8s.io/apimachinery/pkg/types" + + "github.com/ConfigButler/gitops-reverser/internal/git" + "github.com/ConfigButler/gitops-reverser/internal/queue" +) + +// These tests drive the REAL index through the real resolver, rather than a fake lookup, because +// the thing worth proving is the wiring between them: the index can only reach its collection tiers +// if the resolver hands it a namespace and the object's labels, and no fake would notice their +// absence. The index's own tier policy is proven in internal/queue; what is proven here is that a +// watch event can actually get there. + +const indexTestRoute = "prod-eu-1" + +var configmapsResolverGVR = schema.GroupVersionResource{Version: "v1", Resource: "configmaps"} + +// newIndexResolver builds a resolver over an empty in-process index and returns both. +func newIndexResolver(t *testing.T, grace time.Duration) (AuthorResolver, *queue.FactIndex) { + t.Helper() + index := queue.NewFactIndex(queue.FactIndexConfig{Log: logr.Discard()}) + return NewAuthorResolver(index, grace, logr.Discard()), index +} + +// applyFact delivers one fact the way the follower would: as an entry on the type's stream. +func applyFact(index *queue.FactIndex, fact queue.AuthorFact) { + index.Apply(context.Background(), queue.FactEntry{ + Key: queue.FactStreamKeyFor(indexTestRoute, configmapsResolverGVR.GroupResource()), + Facts: []queue.AuthorFact{fact}, + }) +} + +// collectionFact is one `kubectl delete configmaps -n team-a -l app=web` as the receiver publishes +// it: one fact about the COLLECTION, carrying the selector the request URI expressed. uids is the +// set the API server said it deleted, and nil is the body-less case. +func collectionFact(selector string, uids []string) queue.AuthorFact { + return queue.AuthorFact{ + Namespace: "team-a", + Author: "alice", + Email: "alice@example.com", + Verb: "deletecollection", + LabelSelector: selector, + UIDs: uids, + StageTimestamp: time.Now().UTC().Format(time.RFC3339Nano), + } +} + +// removalQuery is a DELETE watch event for one object the collection covered. It is not +// exact-capable: a removal's resourceVersion is never the one the write produced. +func removalQuery(uid string, labels map[string]string) AuthorQuery { + return AuthorQuery{ + AuditRoute: indexTestRoute, + GVR: configmapsResolverGVR, + UID: k8stypes.UID(uid), + ResourceVersion: "9999", + Namespace: "team-a", + Labels: labels, + ExactCapable: false, + } +} + +// TestAuthorResolver_CollectionDeleteResolvesByUIDMembership is the precise half of the collection +// join. When the API server sent a response body, the fact carries the uid set it named, and +// membership carries no over-attribution risk at all: either this object was in the set, or it was +// not. +func TestAuthorResolver_CollectionDeleteResolvesByUIDMembership(t *testing.T) { + resolver, index := newIndexResolver(t, 0) + applyFact(index, collectionFact("", []string{"uid-1", "uid-2"})) + + ui, outcome := resolver.ResolveAuthor(context.Background(), removalQuery("uid-1", nil)) + + require.Equal(t, git.AttributionResolved, outcome) + assert.Equal(t, "alice", ui.Username) + + // An object the collection did not cover falls through to the scope tier, which a no-selector + // collection also accepts — the point of the uid tier is precision, not exclusion. + _, outcome = resolver.ResolveAuthor(context.Background(), removalQuery("uid-elsewhere", nil)) + assert.Equal(t, git.AttributionResolved, outcome) +} + +// TestAuthorResolver_BodylessCollectionDeleteResolvesByScope is the case the deleted expander gave +// up on entirely. A truncated, aggregated, or metadata-only deletecollection carries no response +// body, so there is no uid set to join — and every removal in the collection used to ship +// committer-authored. Scope matching resolves it from what the audit event actually said: the type, +// the namespace, and the selector the actor expressed. +// +// A production cluster is the one MOST likely to hit this, because +// --audit-webhook-truncate-enabled drops bodies from oversized events and the ten-thousand-object +// collection delete is exactly the oversized event. +func TestAuthorResolver_BodylessCollectionDeleteResolvesByScope(t *testing.T) { + resolver, index := newIndexResolver(t, 0) + applyFact(index, collectionFact("app=web", nil)) + + ui, outcome := resolver.ResolveAuthor(context.Background(), + removalQuery("uid-1", map[string]string{"app": "web"})) + + require.Equal(t, git.AttributionResolved, outcome, + "a body-less collection delete must resolve by scope, not degrade to the committer") + assert.Equal(t, "alice", ui.Username) + assert.Equal(t, "alice@example.com", ui.Email) +} + +// An object the selector does not accept was not part of what the actor asked to delete, so it must +// NOT be credited to them. Scope matching is the weakest evidence the join has, and naming the +// wrong human is worse than naming nobody. +func TestAuthorResolver_CollectionScopeDoesNotClaimUnselectedObjects(t *testing.T) { + resolver, index := newIndexResolver(t, 0) + applyFact(index, collectionFact("app=web", nil)) + + _, outcome := resolver.ResolveAuthor(context.Background(), + removalQuery("uid-1", map[string]string{"app": "db"})) + + assert.Equal(t, git.AttributionUnresolved, outcome) +} + +// A collection in one namespace says nothing about an object in another, even on the same route and +// type. The namespace has to travel with the query for that to be decidable at all. +func TestAuthorResolver_CollectionScopeIsNamespaceBound(t *testing.T) { + resolver, index := newIndexResolver(t, 0) + applyFact(index, collectionFact("", nil)) + + query := removalQuery("uid-1", nil) + query.Namespace = "team-b" + _, outcome := resolver.ResolveAuthor(context.Background(), query) + + assert.Equal(t, git.AttributionUnresolved, outcome) +} + +// TestAuthorResolver_RouteIsolatesOtherwiseIdenticalFacts proves the partition survives the whole +// resolver path, not only the index's own keys. Two clusters can hold objects with the same uid and +// the same resourceVersion, and a fact from one must never name the author on the other. +func TestAuthorResolver_RouteIsolatesOtherwiseIdenticalFacts(t *testing.T) { + resolver, index := newIndexResolver(t, 0) + index.Apply(context.Background(), queue.FactEntry{ + Key: queue.FactStreamKeyFor("prod-us-1", configmapsResolverGVR.GroupResource()), + Facts: []queue.AuthorFact{{Namespace: "team-a", UID: "uid-1", + ResourceVersion: "101", Author: "carol", Verb: "update", + }}, + }) + + query := AuthorQuery{ + AuditRoute: indexTestRoute, GVR: configmapsResolverGVR, + UID: "uid-1", ResourceVersion: "101", Namespace: "team-a", ExactCapable: true, + } + _, outcome := resolver.ResolveAuthor(context.Background(), query) + assert.Equal(t, git.AttributionUnresolved, outcome, + "a fact from another cluster's audit route must not name this object's author") + + query.AuditRoute = "prod-us-1" + ui, outcome := resolver.ResolveAuthor(context.Background(), query) + require.Equal(t, git.AttributionResolved, outcome) + assert.Equal(t, "carol", ui.Username) +} + +// TestAuthorResolver_WaitsForAFactDeliveredDuringTheGrace is the change's whole premise, end to +// end. Audit delivery is batched by the API server while the watch is streamed, so the fact +// reliably arrives AFTER the event that needs it. The resolver no longer polls for it: it registers +// a waiter, finds nothing, and is woken by the goroutine applying the fact. +func TestAuthorResolver_WaitsForAFactDeliveredDuringTheGrace(t *testing.T) { + resolver, index := newIndexResolver(t, 2*time.Second) + + go func() { + time.Sleep(30 * time.Millisecond) + applyFact(index, queue.AuthorFact{Namespace: "team-a", UID: "uid-1", + ResourceVersion: "101", Author: "bob", Verb: "update", + }) + }() + + start := time.Now() + ui, outcome := resolver.ResolveAuthor(context.Background(), AuthorQuery{ + AuditRoute: indexTestRoute, GVR: configmapsResolverGVR, + UID: "uid-1", ResourceVersion: "101", Namespace: "team-a", ExactCapable: true, + }) + + require.Equal(t, git.AttributionResolved, outcome) + assert.Equal(t, "bob", ui.Username) + assert.Less(t, time.Since(start), 2*time.Second, + "the waiter is woken by the fact, not by the grace deadline expiring") +} + +// A cancelled context ends the wait immediately rather than holding the watch shard for the whole +// grace window. The shard is single-threaded, so a wait that ignored shutdown would stall it. +func TestAuthorResolver_CancelledContextEndsTheWait(t *testing.T) { + resolver, _ := newIndexResolver(t, time.Minute) + ctx, cancel := context.WithCancel(context.Background()) + cancel() + + start := time.Now() + _, outcome := resolver.ResolveAuthor(ctx, AuthorQuery{ + AuditRoute: indexTestRoute, GVR: configmapsResolverGVR, + UID: "uid-1", ResourceVersion: "101", Namespace: "team-a", ExactCapable: true, + }) + + assert.Equal(t, git.AttributionUnresolved, outcome) + assert.Less(t, time.Since(start), 5*time.Second) +} diff --git a/internal/watch/author_resolver_test.go b/internal/watch/author_resolver_test.go index 52b36537..43bf0e7c 100644 --- a/internal/watch/author_resolver_test.go +++ b/internal/watch/author_resolver_test.go @@ -18,26 +18,50 @@ import ( "github.com/ConfigButler/gitops-reverser/internal/telemetry" ) -// fakeLookup returns fact/ok after `hitAfter` calls; calls counts invocations and -// lastExactCapable records the event-kind flag of the most recent lookup. +// fakeLookup stands in for the fact index. The resolver makes exactly ONE call now — the waiting +// belongs to the index, which is the only thing that knows when a fact arrives — so lateness is +// modelled as a delay before the resolution is returned rather than as a number of retries. type fakeLookup struct { - resolution queue.AuthorResolution - hitAfter int - calls int - lastExactCapable bool - lastProvider string + resolution queue.AuthorResolution + // availableAfter is how long the fact takes to arrive. Longer than the grace it is handed means + // it never arrives in time, which is what the index reports as absent. + availableAfter time.Duration + calls int + lastQuery queue.FactQuery + lastGrace time.Duration } -func (f *fakeLookup) LookupAuthorResolution( - _ context.Context, providerName string, _ schema.GroupVersionResource, _ k8stypes.UID, _ string, exactCapable bool, +func (f *fakeLookup) Await( + ctx context.Context, + query queue.FactQuery, + grace time.Duration, ) queue.AuthorResolution { f.calls++ - f.lastExactCapable = exactCapable - f.lastProvider = providerName - if f.calls >= f.hitAfter { - return f.resolution + f.lastQuery = query + f.lastGrace = grace + if f.availableAfter > grace { + return queue.AuthorResolution{Result: queue.AttributionAbsent} + } + if f.availableAfter > 0 { + select { + case <-ctx.Done(): + return queue.AuthorResolution{Result: queue.AttributionAbsent} + case <-time.After(f.availableAfter): + } + } + return f.resolution +} + +// resolverQuery is the ordinary object-write query these tests drive the resolver with. +func resolverQuery(route, uid, rv string, exactCapable bool) AuthorQuery { + return AuthorQuery{ + AuditRoute: route, + GVR: resolverGVR, + UID: k8stypes.UID(uid), + ResourceVersion: rv, + Namespace: "team-a", + ExactCapable: exactCapable, } - return queue.AuthorResolution{Result: queue.AttributionAbsent} } var resolverGVR = schema.GroupVersionResource{Group: "apps", Version: "v1", Resource: "deployments"} @@ -46,60 +70,144 @@ func TestAuthorResolver_HumanHit(t *testing.T) { lookup := &fakeLookup{ resolution: queue.AuthorResolution{ Fact: queue.AuthorFact{Author: "alice", Email: "a@x.io"}, - Result: queue.AttributionExactUser, + Result: queue.AttributionExact, }, - hitAfter: 1, } r := NewAuthorResolver(lookup, DefaultAttributionGraceWindow, logr.Discard()) - ui, outcome := r.ResolveAuthor(context.Background(), "prod-eu-1", resolverGVR, "uid-1", "101", true) + ui, outcome := r.ResolveAuthor(context.Background(), resolverQuery("prod-eu-1", "uid-1", "101", true)) require.Equal(t, git.AttributionResolved, outcome) assert.Equal(t, "alice", ui.Username) assert.Equal(t, "a@x.io", ui.Email) assert.Equal(t, 1, lookup.calls) - assert.True(t, lookup.lastExactCapable, "an ADDED/MODIFIED event is exact-capable") + assert.True(t, lookup.lastQuery.ExactCapable, "an ADDED/MODIFIED event is exact-capable") } func TestAuthorResolver_ServiceAccountIsNamed(t *testing.T) { reader, err := telemetry.InitTestExporter() require.NoError(t, err) - // A matched service account is always named by its own username — never collapsed - // to the committer — and the resolution is recorded as exact_serviceaccount. + // A matched service account is always named by its own username — never collapsed to the + // committer — and the tier and the actor kind are recorded as two separate labels. They used to + // be one value, exact_serviceaccount, which made "how many exact resolutions" a sum of two + // series and made the actor kind unaskable of any other tier. sa := "system:serviceaccount:flux-system:kustomize-controller" lookup := &fakeLookup{ resolution: queue.AuthorResolution{ - Fact: queue.AuthorFact{Author: sa, IsServiceAccount: true}, - Result: queue.AttributionExactServiceAccount, + Fact: queue.AuthorFact{Author: sa}, + Result: queue.AttributionExact, }, - hitAfter: 1, } r := NewAuthorResolver(lookup, DefaultAttributionGraceWindow, logr.Discard()) - ui, outcome := r.ResolveAuthor(context.Background(), "prod-eu-1", resolverGVR, "uid-1", "101", true) + ui, outcome := r.ResolveAuthor(context.Background(), resolverQuery("prod-eu-1", "uid-1", "101", true)) require.Equal(t, git.AttributionResolved, outcome, "a matched service account is named, not collapsed to the committer") assert.Equal(t, sa, ui.Username) count, ok := telemetry.CollectInt64Sum(reader, "gitopsreverser_attribution_resolutions_total", - map[string]string{"result": string(queue.AttributionExactServiceAccount)}) + map[string]string{ + "tier": string(queue.AttributionExact), + "actor_kind": string(queue.ActorKindServiceAccount), + "resource": "deployments", + }) require.True(t, ok) assert.Equal(t, int64(1), count) + // The wait histogram carries the tier and the KIND OF EVENT instead of the actor: an ADDED or + // MODIFIED event is a write, and a write does not hold a fallback and keep waiting. waitCount, ok := telemetry.CollectHistogramCount(reader, "gitopsreverser_attribution_resolution_wait_seconds", - map[string]string{"result": string(queue.AttributionExactServiceAccount)}) + map[string]string{ + "tier": string(queue.AttributionExact), + "event_kind": attributionEventKindWrite, + }) require.True(t, ok) assert.Equal(t, uint64(1), waitCount) } +// TestAuthorResolver_UserAndServiceAccountShareOneTier proves the split does what it was for: one +// tier series, two actor kinds under it. +func TestAuthorResolver_UserAndServiceAccountShareOneTier(t *testing.T) { + reader, err := telemetry.InitTestExporter() + require.NoError(t, err) + + for _, author := range []string{"alice", "system:serviceaccount:flux-system:kustomize-controller"} { + lookup := &fakeLookup{ + resolution: queue.AuthorResolution{ + Fact: queue.AuthorFact{Author: author}, + Result: queue.AttributionExact, + }, + } + r := NewAuthorResolver(lookup, DefaultAttributionGraceWindow, logr.Discard()) + _, outcome := r.ResolveAuthor(context.Background(), resolverQuery("prod-eu-1", "uid-1", "101", true)) + require.Equal(t, git.AttributionResolved, outcome) + } + + tierTotal, ok := telemetry.CollectInt64Sum(reader, "gitopsreverser_attribution_resolutions_total", + map[string]string{"tier": string(queue.AttributionExact)}) + require.True(t, ok) + assert.Equal(t, int64(2), tierTotal, "counting one tier is one selector, not a sum of two series") + + for kind, want := range map[queue.ActorKind]int64{ + queue.ActorKindUser: 1, + queue.ActorKindServiceAccount: 1, + // Neither resolution was a miss, so nothing was named "none" — the third value is asserted + // absent rather than left unstated. + queue.ActorKindNone: 0, + } { + got, found := telemetry.CollectInt64Sum(reader, "gitopsreverser_attribution_resolutions_total", + map[string]string{"tier": string(queue.AttributionExact), "actor_kind": string(kind)}) + require.Equal(t, want > 0, found, "actor kind %q", kind) + assert.Equal(t, want, got, "actor kind %q", kind) + } +} + +// TestAuthorResolver_RemovalWaitIsItsOwnSeries covers the distinction --author-attribution-grace is +// tuned from: an absent write and an absent removal used to land in one histogram series, and only +// the removal sits out the whole grace window. +func TestAuthorResolver_RemovalWaitIsItsOwnSeries(t *testing.T) { + reader, err := telemetry.InitTestExporter() + require.NoError(t, err) + + lookup := &fakeLookup{resolution: queue.AuthorResolution{Result: queue.AttributionAbsent}} + r := NewAuthorResolver(lookup, 0, logr.Discard()) + + _, outcome := r.ResolveAuthor(context.Background(), resolverQuery("prod-eu-1", "uid-1", "999", false)) + require.Equal(t, git.AttributionUnresolved, outcome) + + removals, ok := telemetry.CollectHistogramCount(reader, "gitopsreverser_attribution_resolution_wait_seconds", + map[string]string{ + "tier": string(queue.AttributionAbsent), + "event_kind": attributionEventKindRemoval, + }) + require.True(t, ok) + assert.Equal(t, uint64(1), removals) + + _, writes := telemetry.CollectHistogramCount(reader, "gitopsreverser_attribution_resolution_wait_seconds", + map[string]string{ + "tier": string(queue.AttributionAbsent), + "event_kind": attributionEventKindWrite, + }) + assert.False(t, writes, "a removal's wait must not be counted as a write's") + + // An unmatched resolution names nobody, and says so rather than leaving the label off. + absent, ok := telemetry.CollectInt64Sum(reader, "gitopsreverser_attribution_resolutions_total", + map[string]string{ + "tier": string(queue.AttributionAbsent), + "actor_kind": string(queue.ActorKindNone), + }) + require.True(t, ok) + assert.Equal(t, int64(1), absent) +} + func TestAuthorResolver_MissExpiresToUnresolved(t *testing.T) { - lookup := &fakeLookup{resolution: queue.AuthorResolution{Result: queue.AttributionAbsent}, hitAfter: 1000} + lookup := &fakeLookup{resolution: queue.AuthorResolution{Result: queue.AttributionAbsent}} r := NewAuthorResolver(lookup, 0, logr.Discard()) // A zero grace does a single lookup and, on a miss, reports UNRESOLVED — attribution ran // and did not name anyone. It is deliberately not NotAttempted, which would claim // attribution was never switched on. There is no miss-marker write-back. - ui, outcome := r.ResolveAuthor(context.Background(), "prod-eu-1", resolverGVR, "uid-1", "101", true) + ui, outcome := r.ResolveAuthor(context.Background(), resolverQuery("prod-eu-1", "uid-1", "101", true)) assert.Equal(t, git.AttributionUnresolved, outcome) assert.Empty(t, ui.Username, "an unresolved attribution names nobody") assert.Equal(t, 1, lookup.calls) @@ -109,31 +217,30 @@ func TestAuthorResolver_DeleteEventIsNotExactCapable(t *testing.T) { lookup := &fakeLookup{ resolution: queue.AuthorResolution{ Fact: queue.AuthorFact{Author: "alice"}, - Result: queue.AttributionWeak, + Result: queue.AttributionLatest, }, - hitAfter: 1, } r := NewAuthorResolver(lookup, DefaultAttributionGraceWindow, logr.Discard()) - _, outcome := r.ResolveAuthor(context.Background(), "prod-eu-1", resolverGVR, "uid-1", "999", false) + _, outcome := r.ResolveAuthor(context.Background(), resolverQuery("prod-eu-1", "uid-1", "999", false)) require.Equal(t, git.AttributionResolved, outcome) - assert.False(t, lookup.lastExactCapable, "a removal event may consult the /last pointer") + assert.False(t, lookup.lastQuery.ExactCapable, "a removal event may consult the weaker tiers") } func TestAuthorResolver_WaitsThroughGraceWindowForLateFact(t *testing.T) { lookup := &fakeLookup{ resolution: queue.AuthorResolution{ Fact: queue.AuthorFact{Author: "bob"}, - Result: queue.AttributionExactUser, + Result: queue.AttributionExact, }, - hitAfter: 3, + availableAfter: 50 * time.Millisecond, } r := NewAuthorResolver(lookup, 2*time.Second, logr.Discard()) - ui, outcome := r.ResolveAuthor(context.Background(), "prod-eu-1", resolverGVR, "uid-1", "101", true) + ui, outcome := r.ResolveAuthor(context.Background(), resolverQuery("prod-eu-1", "uid-1", "101", true)) require.Equal(t, git.AttributionResolved, outcome) assert.Equal(t, "bob", ui.Username) - assert.GreaterOrEqual(t, lookup.calls, 3) + assert.Equal(t, 1, lookup.calls, "the resolver asks once and the index does the waiting") } // A nil lookup is configured-author mode: attribution was never switched on, so the outcome @@ -142,7 +249,7 @@ func TestAuthorResolver_WaitsThroughGraceWindowForLateFact(t *testing.T) { func TestAuthorResolver_NilLookupIsNotAttempted(t *testing.T) { r := NewAuthorResolver(nil, DefaultAttributionGraceWindow, logr.Discard()) - ui, outcome := r.ResolveAuthor(context.Background(), "prod-eu-1", resolverGVR, "uid-1", "101", true) + ui, outcome := r.ResolveAuthor(context.Background(), resolverQuery("prod-eu-1", "uid-1", "101", true)) assert.Equal(t, git.AttributionNotAttempted, outcome, "attribution that was never enabled has not failed — the committer legitimately authors") @@ -151,19 +258,37 @@ func TestAuthorResolver_NilLookupIsNotAttempted(t *testing.T) { // A fact that exists but carries no author is also unresolved, not not-attempted: attribution // ran, found something, and still could not name anyone. +// +// The publish gate makes this unreachable in production — AuthorFactFromEvent refuses an event +// whose user cannot be resolved, and counts it as no_attribution_fact — so this pins the DEFENSIVE +// branch, and with it the invariant every coverage query rests on: the tier says which evidence +// answered, and if that evidence names nobody the metrics say so on the actor_kind label rather +// than quietly counting it as a named actor. func TestAuthorResolver_AuthorlessFactIsUnresolved(t *testing.T) { + reader, err := telemetry.InitTestExporter() + require.NoError(t, err) + lookup := &fakeLookup{ resolution: queue.AuthorResolution{ Fact: queue.AuthorFact{Author: ""}, - Result: queue.AttributionExactUser, + Result: queue.AttributionExact, }, - hitAfter: 1, } r := NewAuthorResolver(lookup, DefaultAttributionGraceWindow, logr.Discard()) - _, outcome := r.ResolveAuthor(context.Background(), "prod-eu-1", resolverGVR, "uid-1", "101", true) + _, outcome := r.ResolveAuthor(context.Background(), resolverQuery("prod-eu-1", "uid-1", "101", true)) assert.Equal(t, git.AttributionUnresolved, outcome) + + // The tier is the one that matched, and the actor kind is none — so a reader can tell this apart + // from a resolution that named somebody, which reading coverage off the tier alone cannot. + named, ok := telemetry.CollectInt64Sum(reader, "gitopsreverser_attribution_resolutions_total", + map[string]string{ + "tier": string(queue.AttributionExact), + "actor_kind": string(queue.ActorKindNone), + }) + require.True(t, ok, "an authorless match must not be recorded as a named actor") + assert.Equal(t, int64(1), named) } // TestAuthorResolver_WarnsOnceForARouteThatNeverResolves drives the whole resolver, not just the @@ -175,15 +300,14 @@ func TestAuthorResolver_AuthorlessFactIsUnresolved(t *testing.T) { // unnoticed until an explicit unresolved-author placeholder made it visible in Git. func TestAuthorResolver_WarnsOnceForARouteThatNeverResolves(t *testing.T) { // hitAfter is beyond any call this test makes, so every lookup misses. - lookup := &fakeLookup{hitAfter: 1 << 30} + lookup := &fakeLookup{resolution: queue.AuthorResolution{Result: queue.AttributionAbsent}} resolver := NewAuthorResolver(lookup, 0, logr.Discard()) concrete, ok := resolver.(*attributionResolver) require.True(t, ok) const route = "srcns-delegating" for range attributionUnresolvedWarnThreshold { - _, outcome := resolver.ResolveAuthor( - context.Background(), route, resolverGVR, "uid-1", "101", true) + _, outcome := resolver.ResolveAuthor(context.Background(), resolverQuery(route, "uid-1", "101", true)) require.Equal(t, git.AttributionUnresolved, outcome) } @@ -195,10 +319,10 @@ func TestAuthorResolver_WarnsOnceForARouteThatNeverResolves(t *testing.T) { other := &fakeLookup{ resolution: queue.AuthorResolution{ Fact: queue.AuthorFact{Author: "alice"}, - Result: queue.AttributionExactUser, + Result: queue.AttributionExact, }, } healthy := NewAuthorResolver(other, 0, logr.Discard()) - _, outcome := healthy.ResolveAuthor(context.Background(), "default", resolverGVR, "uid-2", "1", true) + _, outcome := healthy.ResolveAuthor(context.Background(), resolverQuery("default", "uid-2", "1", true)) assert.Equal(t, git.AttributionResolved, outcome) } diff --git a/internal/watch/manager.go b/internal/watch/manager.go index b124a063..0dfbd3ad 100644 --- a/internal/watch/manager.go +++ b/internal/watch/manager.go @@ -26,6 +26,7 @@ import ( v1alpha3 "github.com/ConfigButler/gitops-reverser/api/v1alpha3" "github.com/ConfigButler/gitops-reverser/internal/git" + "github.com/ConfigButler/gitops-reverser/internal/queue" "github.com/ConfigButler/gitops-reverser/internal/rulestore" "github.com/ConfigButler/gitops-reverser/internal/telemetry" "github.com/ConfigButler/gitops-reverser/internal/types" @@ -65,6 +66,13 @@ type Manager struct { // joining the audit attribution index (RV/UID match, bounded grace window). Nil // is configured-author mode (no audit/Redis): every event commits as the committer. AuthorResolver AuthorResolver + // FactStreams is the process-wide subscription set the attribution fact follower reads. Each + // running target watch holds one reference on the (audit route, group/resource) it covers, so + // the process follows a type while at least one watch needs it and stops following it when the + // last one goes away — which is the whole point of the per-type fan-out: facts for a type nobody + // watches are appended and never received. Nil is configured-author mode: no follower runs, so + // no subscription is taken. + FactStreams *queue.FactStreamSet // WatchCursorStore optionally persists per-watch resourceVersion cursors so // reconnects can resume without replaying the full type snapshot. WatchCursorStore CursorStore diff --git a/internal/watch/target_watch.go b/internal/watch/target_watch.go index d495e0c6..b44a8d4b 100644 --- a/internal/watch/target_watch.go +++ b/internal/watch/target_watch.go @@ -24,6 +24,7 @@ import ( configv1alpha3 "github.com/ConfigButler/gitops-reverser/api/v1alpha3" "github.com/ConfigButler/gitops-reverser/internal/git" "github.com/ConfigButler/gitops-reverser/internal/manifestanalyzer" + "github.com/ConfigButler/gitops-reverser/internal/queue" "github.com/ConfigButler/gitops-reverser/internal/sanitize" "github.com/ConfigButler/gitops-reverser/internal/types" ) @@ -298,6 +299,14 @@ func (m *Manager) runTargetWatch( key targetWatchKey, ops OperationSet, ) { + // Follow this type's attribution facts for exactly as long as this watch runs. The release is + // idempotent, so calling it here and on any error path below cannot unfollow a type another + // watch still needs. Acquiring before the first session opens is deliberate: the follower reads + // a newly followed stream from the TTL horizon, so the index is warm with the whole retention + // window before the first event of this watch needs an author. + releaseFacts := m.followFactsForWatch(gitDest, key) + defer releaseFacts() + // A target-watch declaration defines the fidelity epoch. Its first session must replay even // when a durable cursor exists: a replacement can add a sibling scope, and resuming an unchanged // scope would otherwise leave that scope pending in the new epoch forever. Later reconnects may @@ -320,6 +329,21 @@ func (m *Manager) runTargetWatch( } } +// followFactsForWatch takes one reference on the attribution fact stream for the (audit route, +// group/resource) this watch covers, and returns the release for it. It is a no-op returning a +// no-op in configured-author mode, where no follower runs and no subscription is meaningful. +// +// The route rather than the cluster id is what the stream is keyed on, for the same reason the join +// is: an API server posts audit under ONE route, so several ClusterProviders naming one cluster all +// declare that route and share its facts. +func (m *Manager) followFactsForWatch(gitDest types.ResourceReference, key targetWatchKey) func() { + if m.FactStreams == nil { + return func() {} + } + route := m.auditRouteForCluster(m.clusterIDForGitTarget(gitDest)) + return m.FactStreams.Acquire(queue.FactStreamKeyFor(route, key.GVR.GroupResource())) +} + func (m *Manager) targetWatchReplayAndStream( ctx context.Context, log logr.Logger, @@ -771,9 +795,21 @@ func (m *Manager) attachAuthor( // that same route and joins the same facts. Keying the read on the provider name instead was // the bug this indirection exists to prevent, and a fact from cluster A still cannot name the // author of an object watched on cluster B, because their routes differ. - userInfo, outcome := m.AuthorResolver.ResolveAuthor( - ctx, m.auditRouteForCluster(event.SourceCluster), gvr, u.GetUID(), u.GetResourceVersion(), exactCapable, - ) + // + // Namespace and labels ride along for the collection tier: a removal caused by a + // deletecollection the API server sent no response body for is joined by SCOPE — same type and + // namespace, the request's selector accepting these labels, within the collection window — which + // is the case the deleted expander gave up on entirely. + userInfo, outcome := m.AuthorResolver.ResolveAuthor(ctx, AuthorQuery{ + AuditRoute: m.auditRouteForCluster(event.SourceCluster), + GVR: gvr, + UID: u.GetUID(), + ResourceVersion: u.GetResourceVersion(), + Namespace: u.GetNamespace(), + Labels: u.GetLabels(), + Name: u.GetName(), + ExactCapable: exactCapable, + }) // Stamp the outcome even when no actor was named: an unresolved attribution is a fact the // writer, the author_kind metric, and CommitRequest matching all need. Leaving it at the // zero value would say "attribution was never attempted", which is exactly the conflation diff --git a/internal/watch/target_watch_test.go b/internal/watch/target_watch_test.go index 18cca748..155e1ebf 100644 --- a/internal/watch/target_watch_test.go +++ b/internal/watch/target_watch_test.go @@ -208,9 +208,8 @@ func TestRouteLiveTargetWatchEvent_AttributesAuthorFromResolver(t *testing.T) { &fakeLookup{ resolution: queue.AuthorResolution{ Fact: queue.AuthorFact{Author: "alice", Email: "alice@example.com"}, - Result: queue.AttributionExactUser, + Result: queue.AttributionExact, }, - hitAfter: 1, }, time.Second, logr.Discard(), ), @@ -843,3 +842,54 @@ func (f *fakeWatchCursorStore) lastLookedUpUID() string { defer f.mu.Unlock() return f.lookedUpUID } + +// TestFollowFactsForWatch_ReferenceCountsPerTypeAndRoute proves the fan-out is driven by the +// watches themselves. A type is followed while at least one watch covers it and unfollowed when the +// last one goes away, so facts for a type nobody watches are appended to a stream and never +// received — which is the only reason publishing per type is cheaper than publishing everything. +// +// The reference count is the part that has to be right. Several WatchRules and several GitTargets +// routinely cover one type; a plain set would unfollow on the first watch to stop and silently kill +// attribution for the ones still running. +func TestFollowFactsForWatch_ReferenceCountsPerTypeAndRoute(t *testing.T) { + manager := &Manager{FactStreams: queue.NewFactStreamSet()} + gitDest := types.NewResourceReference("target", "default") + other := types.NewResourceReference("second-target", "default") + + releaseFirst := manager.followFactsForWatch(gitDest, targetWatchKey{GVR: configmapsGVR, Namespace: "apps"}) + require.Equal(t, 1, manager.FactStreams.Len()) + + // A second watch on the SAME type in another namespace, and a second GitTarget on the same + // type, both share one stream: a fact names a write that happened, not a consumer interested + // in it. + releaseSecond := manager.followFactsForWatch(gitDest, targetWatchKey{GVR: configmapsGVR, Namespace: "ops"}) + releaseThird := manager.followFactsForWatch(other, targetWatchKey{GVR: configmapsGVR}) + assert.Equal(t, 1, manager.FactStreams.Len(), "one type is one stream, however many watches cover it") + + // A different type is a different stream. + releaseDeployments := manager.followFactsForWatch(gitDest, targetWatchKey{GVR: resolverGVR}) + assert.Equal(t, 2, manager.FactStreams.Len()) + + releaseFirst() + releaseFirst() // idempotent: a watch torn down on both an error path and its deferred cleanup + assert.Equal(t, 2, manager.FactStreams.Len(), "the type stays followed while another watch needs it") + + releaseSecond() + releaseThird() + assert.Equal(t, 1, manager.FactStreams.Len(), "the last watch on a type unfollows it") + + releaseDeployments() + assert.Zero(t, manager.FactStreams.Len()) +} + +// Configured-author mode has no follower and no index, so a watch takes no subscription. The nil +// check is what keeps attribution optional rather than merely disabled. +func TestFollowFactsForWatch_IsANoOpWithoutAttribution(t *testing.T) { + manager := &Manager{} + release := manager.followFactsForWatch( + types.NewResourceReference("target", "default"), + targetWatchKey{GVR: configmapsGVR}, + ) + require.NotNil(t, release, "the release must be safe to defer even with attribution off") + release() +} diff --git a/internal/webhook/audit_fact_publish_test.go b/internal/webhook/audit_fact_publish_test.go index 661a17d5..5fdb6832 100644 --- a/internal/webhook/audit_fact_publish_test.go +++ b/internal/webhook/audit_fact_publish_test.go @@ -6,13 +6,16 @@ import ( "context" "errors" "net/http" + "os" "sync" "testing" + "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" "k8s.io/apimachinery/pkg/runtime/schema" "github.com/ConfigButler/gitops-reverser/internal/queue" + "github.com/ConfigButler/gitops-reverser/internal/telemetry" ) // factAppend is one PublishFacts call: one stream, one entry, however many facts the request @@ -25,9 +28,13 @@ type factAppend struct { // fakeFactPublisher records every append, so a test can count them. The count is the point: an // apiserver batch over three types must become three appends, not one per event. type fakeFactPublisher struct { - mu sync.Mutex - err error - appends []factAppend + mu sync.Mutex + err error + // failAfter makes the publisher start failing once it has appended this many batches, so a test + // can drive a PARTIAL publication: the shape that decides whether an earlier stream's events are + // reported as lost. Zero means it never fails on its own. + failAfter int + appends []factAppend } func (p *fakeFactPublisher) PublishFacts(_ context.Context, key queue.FactStreamKey, facts []queue.AuthorFact) error { @@ -36,6 +43,9 @@ func (p *fakeFactPublisher) PublishFacts(_ context.Context, key queue.FactStream if p.err != nil { return p.err } + if p.failAfter > 0 && len(p.appends) >= p.failAfter { + return errors.New("transport down") + } p.appends = append(p.appends, factAppend{key: key, facts: facts}) return nil } @@ -92,12 +102,12 @@ func TestAuditHandler_OneRequestOverThreeTypesBecomesThreeAppends(t *testing.T) require.Equal(t, queue.FactStreamKeyFor("prod-eu-1", schema.GroupResource{Group: "apps", Resource: "deployments"}), appends[0].key) - require.Equal(t, []string{"web", "api"}, factNames(appends[0].facts), + require.Equal(t, []string{"uid-web", "uid-api"}, factUIDs(appends[0].facts), "a group keeps the order its events arrived in") require.Equal(t, queue.FactStreamKeyFor("prod-eu-1", schema.GroupResource{Resource: "configmaps"}), appends[1].key) - require.Equal(t, []string{"config", "other"}, factNames(appends[1].facts)) + require.Equal(t, []string{"uid-config", "uid-other"}, factUIDs(appends[1].facts)) require.Equal(t, queue.FactStreamKeyFor("prod-eu-1", schema.GroupResource{Resource: "secrets"}), appends[2].key) - require.Equal(t, []string{"creds"}, factNames(appends[2].facts)) + require.Equal(t, []string{"uid-creds"}, factUIDs(appends[2].facts)) require.Equal(t, "alice", appends[0].facts[0].Author) require.Equal(t, "101", appends[0].facts[0].ResourceVersion) @@ -151,7 +161,7 @@ func TestAuditHandler_NameLessDeleteCollectionPublishesOneFactWithItsSelector(t require.Equal(t, "team-a", fact.Namespace) require.Equal(t, "app=web", fact.LabelSelector, "the selector is the intent the actor expressed") require.Equal(t, []string{"uid-1", "uid-2"}, fact.UIDs, "a body that was there upgrades the join to uid membership") - require.Empty(t, fact.Name, "a collection request names no object") + require.Empty(t, fact.UID, "a collection request names no object") } func TestAuditHandler_PublishFailureIsRetryable(t *testing.T) { @@ -166,8 +176,8 @@ func TestAuditHandler_PublishFailureIsRetryable(t *testing.T) { } func TestAuditHandler_NoPublisherPublishesNothing(t *testing.T) { - recorder := &fakeFactRecorder{} - handler, err := NewAuditHandler(AuditHandlerConfig{FactRecorder: recorder}) + recorder := &fakeFactSink{} + handler, err := NewAuditHandler(AuditHandlerConfig{FactPublisher: recorder}) require.NoError(t, err) // Configured-author mode, and every install that has not wired the stream: the keys are still @@ -177,11 +187,234 @@ func TestAuditHandler_NoPublisherPublishesNothing(t *testing.T) { require.Equal(t, 1, recorder.len()) } -// factNames flattens a batch to the object names it is about. -func factNames(facts []queue.AuthorFact) []string { +// factUIDs flattens a batch to the object uids it is about. The uid is what identifies an object in +// a fact now — the name was dropped from the wire, because no join tier reads it. +func factUIDs(facts []queue.AuthorFact) []string { names := make([]string, 0, len(facts)) for _, fact := range facts { - names = append(names, fact.Name) + names = append(names, fact.UID) } return names } + +// annotatedWriteEvent is one accepted write carrying the audit-route annotation the shared, +// annotation-routed endpoint reads its route from. +func annotatedWriteEvent(auditID, name, user, route string) string { + event := writeEvent(auditID, "", "configmaps", name, user) + return event[:len(event)-1] + `,"annotations":{"` + clusterAnnotation + `":"` + route + `"}}` +} + +// TestAuditHandler_OneBatchFansOutToOneStreamPerRoute is route isolation at the ingress, where it +// first has to hold. A shared audit stream may carry several logical clusters in ONE batch, and two +// clusters routinely hold objects of the same type — so facts that differ only by route must land +// on different streams. Pooling them would let a fact from cluster A name the author of an object +// watched on cluster B, which is the failure the route dimension exists to prevent and which no +// amount of correctness further down the join could undo. +func TestAuditHandler_OneBatchFansOutToOneStreamPerRoute(t *testing.T) { + publisher := &fakeFactPublisher{} + handler, err := NewAuditHandler(AuditHandlerConfig{ + FactPublisher: publisher, + AuditRouteAnnotationKey: clusterAnnotation, + }) + require.NoError(t, err) + + body := eventListBody( + annotatedWriteEvent("a", "config", "alice", "prod-eu-1"), + annotatedWriteEvent("b", "config", "mallory", "prod-us-1"), + annotatedWriteEvent("c", "other", "alice", "prod-eu-1"), + ) + require.Equal(t, http.StatusOK, serveBody(t, handler, http.MethodPost, "/audit-webhook", body).Code) + + appends := publisher.recorded() + require.Len(t, appends, 2, "the same type on two routes is two streams, not one") + + byRoute := map[string][]queue.AuthorFact{} + for _, append := range appends { + require.Equal(t, schema.GroupResource{Resource: "configmaps"}, append.key.GroupResource) + byRoute[append.key.AuditRoute] = append.facts + } + + require.Len(t, byRoute["prod-eu-1"], 2) + assert.Equal(t, []string{"alice", "alice"}, factAuthors(byRoute["prod-eu-1"])) + require.Len(t, byRoute["prod-us-1"], 1) + assert.Equal(t, []string{"mallory"}, factAuthors(byRoute["prod-us-1"]), + "the other cluster's actor stays on the other cluster's stream") +} + +// factAuthors names each fact's actor, in the order the batch carried them. +func factAuthors(facts []queue.AuthorFact) []string { + authors := make([]string, 0, len(facts)) + for _, fact := range facts { + authors = append(authors, fact.Author) + } + return authors +} + +// TestAuditHandler_CapturedAggregatedCollectionDeletesPickTheirTier drives the REAL captured audit +// events for a deletecollection on an aggregated type, and asserts which join tier each shape +// leaves reachable. They are recordings from a live apiserver rather than fixtures written to fit +// the code, which is what makes them worth asserting against: the question they answer — does a +// proxied collection delete carry the set it deleted? — is a fact about Kubernetes, not about us. +// +// The answer decides everything downstream. A fact carrying uids joins by MEMBERSHIP, which cannot +// name the wrong actor. A fact without them falls back to SCOPE, which can, and is bounded by the +// namespace, the selector, and a short window instead. The deleted expander had no second tier, so +// every one of the body-less shapes below produced nothing at all and shipped committer-authored. +func TestAuditHandler_CapturedAggregatedCollectionDeletesPickTheirTier(t *testing.T) { + tests := map[string]struct { + fixture string + wantUIDs []string + }{ + // The official aggregation layer PROXIES the request and never decodes the response it + // streamed back, so it audits the request with no body. This is the production shape, and + // the one the scope tier exists for. + "official apiserver sends no body": { + fixture: "testdata/audit-events/audit-deletecollection-official-raw-hollow.json", + }, + "official apiserver, namespace teardown": { + fixture: "testdata/audit-events/audit-deletecollection-official-teardown-hollow.json", + }, + // A body-supplying proxy in front of the extension server DOES return the deleted set, and + // then the join upgrades itself: the uids travel with the fact and membership decides. + "body-supplying proxy returns the deleted set": { + fixture: "testdata/audit-events/audit-deletecollection-proxy-raw-listbody.json", + wantUIDs: []string{ + "e1b076ff-f430-4b82-ad7b-c170c4095fe3", + "5ce03312-bab3-4e72-af37-e0ff893a7b76", + }, + }, + // Not every body is a list: a DeleteOptions echo carries no items, so it degrades to scope + // exactly as an absent body does. Parsing has to tell those apart without failing. + "proxy echoing DeleteOptions carries no items": { + fixture: "testdata/audit-events/audit-deletecollection-proxy-teardown-deleteoptions.json", + }, + } + + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + body, err := os.ReadFile(tc.fixture) + require.NoError(t, err, "the captured recording must be readable") + + publisher := &fakeFactPublisher{} + handler := newPublishingHandler(t, publisher) + w := serveBody(t, handler, http.MethodPost, "/audit-webhook/prod-eu-1", string(body)) + require.Equal(t, http.StatusOK, w.Code) + + appends := publisher.recorded() + require.Len(t, appends, 1, "a collection delete is ONE fact about the collection") + require.Len(t, appends[0].facts, 1) + fact := appends[0].facts[0] + + require.Equal(t, "deletecollection", fact.Verb) + require.Equal(t, schema.GroupResource{Group: "wardle.example.com", Resource: "flunders"}, + appends[0].key.GroupResource, "the type is the stream's identity") + require.NotEmpty(t, fact.Namespace, "the scope tier joins on the namespace, so it must survive") + require.NotEmpty(t, fact.Author) + + if tc.wantUIDs == nil { + require.Empty(t, fact.UIDs, + "no usable body means no uid set, and the join falls back to scope matching") + return + } + require.Equal(t, tc.wantUIDs, fact.UIDs, + "a body that was there upgrades the join to uid membership") + }) + } +} + +// TestAuditHandler_PartialPublishOnlyFailsTheBatchesThatDidNotLand pins what a transport failure +// mid-request means for the per-event outcome census. +// +// Publication is per stream and sequential, so a failure is not all-or-nothing: the batches before +// the failing one HAVE appended, and their facts are in the log whatever happens next. Reporting +// those events as write_error would claim a loss that did not occur, and write_error is the one +// outcome an operator is meant to treat as a real problem. +// +// The request still fails, so the API server retries the whole batch and the landed facts are +// appended again — safe precisely because a fact is keyed data rather than a position in a sequence. +func TestAuditHandler_PartialPublishOnlyFailsTheBatchesThatDidNotLand(t *testing.T) { + reader, err := telemetry.InitTestExporter() + require.NoError(t, err) + + // Two types, so the request produces two stream batches; the publisher fails on the second. + publisher := &fakeFactPublisher{failAfter: 1} + handler := newPublishingHandler(t, publisher) + + body := eventListBody( + writeEvent("a", "apps", "deployments", "web", "alice"), + writeEvent("b", "", "configmaps", "config", "bob"), + ) + w := serveBody(t, handler, http.MethodPost, "/audit-webhook/prod-eu-1", body) + require.Equal(t, http.StatusInternalServerError, w.Code, "the request must fail so delivery is retried") + + appends := publisher.recorded() + require.Len(t, appends, 1, "the first batch landed before the second failed") + require.Equal(t, "uid-web", appends[0].facts[0].UID) + + queued, ok := telemetry.CollectInt64Sum(reader, auditEventsMetric, map[string]string{ + "outcome": "queued", "category": "stored", "resource": "deployments", "verb": "update", + }) + require.True(t, ok) + assert.Equal(t, int64(1), queued, "the event whose stream appended is queued, not lost") + + failed, ok := telemetry.CollectInt64Sum(reader, auditEventsMetric, map[string]string{ + "outcome": "write_error", "category": "error", "resource": "configmaps", "verb": "update", + }) + require.True(t, ok) + assert.Equal(t, int64(1), failed, "only the event whose stream did not append is a write error") +} + +// aggregatedCreateEvent is the shape an aggregated API's CREATE is audited with: the kube-apiserver +// proxies the request and never decodes the response, so the objectRef carries no name — the API +// server assigned it — and there is no body to backfill from. +func aggregatedCreateEvent(auditID, user string) string { + return `{"kind":"Event","level":"Metadata","auditID":"` + auditID + `",` + + `"stage":"ResponseComplete","verb":"create","user":{"username":"` + user + `"},` + + `"requestURI":"/apis/wardle.example.com/v1alpha1/namespaces/team-a/flunders",` + + `"objectRef":{"apiGroup":"wardle.example.com","resource":"flunders","namespace":"team-a",` + + `"apiVersion":"wardle.example.com/v1alpha1"},` + + `"responseStatus":{"code":201}}` +} + +// TestAuditHandler_AnEventThatProducesNoFactIsCountedAsSuchPins the population that used to be +// invisible. An accepted event that can never name an author — here an aggregated-API create, whose +// objectRef carries no name at all — produces no fact, so nothing is appended for it and no watch +// event can ever join it. +// +// It used to be counted `queued`, which claimed an append that was never owed and buried the whole +// population under the busiest value on the counter. It is Dropped rather than Error because +// nothing failed, which also keeps the e2e invariant on category="error" intact. +func TestAuditHandler_AnEventThatProducesNoFactIsCountedAsSuch(t *testing.T) { + reader, err := telemetry.InitTestExporter() + require.NoError(t, err) + + publisher := &fakeFactPublisher{} + handler := newPublishingHandler(t, publisher) + + body := eventListBody( + aggregatedCreateEvent("a", "alice"), + writeEvent("b", "", "configmaps", "config", "bob"), + ) + require.Equal(t, http.StatusOK, serveBody(t, handler, http.MethodPost, "/audit-webhook/prod-eu-1", body).Code) + + appends := publisher.recorded() + require.Len(t, appends, 1, "only the joinable event produces a fact") + require.Equal(t, "configmaps", appends[0].key.GroupResource.Resource) + + noFact, ok := telemetry.CollectInt64Sum(reader, auditEventsMetric, map[string]string{ + "outcome": "no_attribution_fact", "category": "dropped", "resource": "flunders", "verb": "create", + }) + require.True(t, ok, "the event that produced no fact must be counted where the decision is made") + assert.Equal(t, int64(1), noFact) + + // The event beside it is unaffected: it appended, so it is queued. + queued, ok := telemetry.CollectInt64Sum(reader, auditEventsMetric, map[string]string{ + "outcome": "queued", "category": "stored", "resource": "configmaps", + }) + require.True(t, ok) + assert.Equal(t, int64(1), queued) + + // The invariant the e2e suite gates on is untouched: nothing here is an error. + _, anyError := telemetry.CollectInt64Sum(reader, auditEventsMetric, map[string]string{"category": "error"}) + assert.False(t, anyError, "an event that owes no append has not failed") +} diff --git a/internal/webhook/audit_handler.go b/internal/webhook/audit_handler.go index 7d4dcaa5..7fad7eb0 100644 --- a/internal/webhook/audit_handler.go +++ b/internal/webhook/audit_handler.go @@ -39,20 +39,11 @@ const DefaultAuditMaxRequestBodyBytes = int64(10 * 1024 * 1024) // each transition so operators can see ingress is wired up at a glance. type auditHandlerFirsts struct { request sync.Once - factRecorded sync.Once + factPublished sync.Once impersonatedEvent sync.Once unroutableEvent sync.Once } -// AuditFactRecorder stores the minimal author-attribution fact for one accepted, mutating audit -// event under the AUDIT ROUTE it arrived on, so a fact from one cluster never joins a watch event -// from another. It is the only thing the audit webhook does now: watch carries the object body, so -// audit is a pure attribution lookup table. A nil recorder means configured-author mode — the -// handler is not wired at all. -type AuditFactRecorder interface { - RecordFact(ctx context.Context, auditRoute string, event auditv1.Event) error -} - // AuditFactPublisher appends one request's attribution facts to the stream for a // (audit route, group/resource). A nil publisher means the fact stream is not wired. // @@ -68,14 +59,14 @@ type AuditFactPublisher interface { type AuditHandlerConfig struct { // MaxRequestBodyBytes is the maximum accepted HTTP request body size. MaxRequestBodyBytes int64 - // FactRecorder persists the attribution fact for each accepted, mutating event. - // A write failure returns an audit-request error so the API server retries - // delivery; mirrored-resource author attribution depends on these facts. - FactRecorder AuditFactRecorder - // FactPublisher appends the facts one request produced, grouped by stream. It is additive to - // FactRecorder for now: the keys are still written and still read, and the stream is filled - // alongside them until the resolver reads from it instead. + // FactPublisher appends the facts one request produced, grouped by stream. A publish failure + // returns an audit-request error so the API server retries the delivery; mirrored-resource + // author attribution depends on these facts arriving. A nil publisher means configured-author + // mode — the handler is not wired at all. FactPublisher AuditFactPublisher + // CollectionUIDCap is how many object uids a deletecollection fact may carry before the set is + // dropped and the join falls back to scope matching. Zero means queue.DefaultCollectionUIDCap. + CollectionUIDCap int // AuditRouteAnnotationKey enables the bare /audit-webhook endpoint for a SHARED audit stream // that carries several logical clusters: the AUDIT ROUTE is read PER EVENT from this // audit-event annotation, so one batch may fan out to several routes. Empty (the default) means @@ -293,63 +284,121 @@ func (h *AuditHandler) decodeEventList(r *http.Request) (*auditv1.EventList, err // Facts are ACCUMULATED across the whole request and appended once per stream at the end, rather // than written per event. That is what turns one apiserver batch over three types into three // appends. +// Each accepted event's terminal outcome is recorded AFTER the append, not before it, so an event +// is counted queued only once its fact is actually in the log. Recording it up front would leave +// audit_events_total{outcome="write_error"} unreachable and claim delivery for a batch that failed. +// +// Publication is per stream and sequential, so a failure is NOT all-or-nothing: the batches before +// the failing one have appended, and their facts are in the log whatever happens next. Counting +// those events as write_error would claim a loss that did not occur. Only the events whose own +// batch did not land are counted as failures. The API server retries the whole request, so the +// batches that did land are appended again — which is safe precisely because a fact is keyed data +// rather than a position in a sequence. func (h *AuditHandler) processEvents(ctx context.Context, route auditRoute, events []auditv1.Event) error { batches := newFactStreamBatches() + accepted := make([]acceptedFact, 0, len(events)) for i := range events { - accepted, err := h.processEvent(ctx, route, events[i]) - if err != nil { - return err + result := h.processEvent(ctx, route, events[i]) + if !result.accepted { + continue + } + accepted = append(accepted, result) + batches.add(result) + } + appended, err := h.publishFactBatches(ctx, batches) + h.recordAcceptedOutcomes(ctx, accepted, appended) + if err != nil { + return err + } + h.logPublishedFacts(accepted) + return nil +} + +// recordAcceptedOutcomes stamps one terminal outcome on every event that passed the accept gate, +// deciding each from whether it produced a fact at all and then from whether its OWN stream +// appended. +// +// An event that produced no fact is not a failure — nothing was owed for it — but it is not queued +// either. It used to be counted queued, which claimed an append that never happened and hid the +// whole population that can never be attributed behind the busiest value on the counter. This is +// the only place it can be counted: no fact exists, so no fact-side counter can ever see it. +func (h *AuditHandler) recordAcceptedOutcomes( + ctx context.Context, + accepted []acceptedFact, + appended map[queue.FactStreamKey]struct{}, +) { + for i := range accepted { + outcome.Record(ctx, accepted[i].event, acceptedOutcome(accepted[i], appended)) + } +} + +// acceptedOutcome is one accepted event's terminal outcome. +func acceptedOutcome(accepted acceptedFact, appended map[queue.FactStreamKey]struct{}) outcome.Outcome { + if !accepted.ok { + return outcome.NoAttributionFact + } + if _, landed := appended[accepted.key]; !landed { + return outcome.WriteError + } + return outcome.Queued +} + +// logPublishedFacts says, once per process and then only at verbosity, which facts were published. +// +// It skips an accepted event that produced no fact. Nothing was appended for it, so calling it +// published would be a line that is simply untrue — and in configured-author mode, where no +// publisher is wired at all, every event is that case. +func (h *AuditHandler) logPublishedFacts(accepted []acceptedFact) { + log := logf.Log.WithName("audit-handler") + for i := range accepted { + if !accepted[i].ok { + continue } - batches.add(accepted) + event := accepted[i].event + h.firsts.factPublished.Do(func() { + log.Info("Published first audit attribution fact", + "auditID", event.AuditID, "verb", event.Verb) + }) + log.V(1).Info("Published audit attribution fact", + "gvr", extractGVR(event), "verb", event.Verb, "auditID", event.AuditID, + "user", effectiveAuditUsername(*event)) } - return h.publishFactBatches(ctx, batches) } -// processEvent applies the intrinsic accept gate, resolves the event's source cluster, records the -// attribution fact for an accepted, mutating event, and returns the fact the request will append. -// A rejected event is recorded with its terminal outcome and dropped; only a fact-store or -// provider-lookup failure returns an error (the API server then retries delivery). +// processEvent applies the intrinsic accept gate, resolves the event's audit route, and reduces an +// accepted, mutating event to the fact the request will append. A rejected event is recorded with +// its terminal outcome here and dropped. It cannot fail: the only failure this path has left is the +// append itself, which happens once per stream after the whole batch is reduced. func (h *AuditHandler) processEvent( ctx context.Context, route auditRoute, event auditv1.Event, -) (acceptedFact, error) { +) acceptedFact { log := logf.Log.WithName("audit-handler") h.logAuditEventReceived(event) if !shouldForwardSubresource(&event) { // A non-/scale subresource (or an unmapped-verb subresource): dropped before recording. outcome.Record(ctx, &event, outcome.NonScaleSubresource) - return acceptedFact{}, nil + return acceptedFact{} } if decision := classifyAuditIngress(&event); !decision.Process { outcome.Record(ctx, &event, outcome.Outcome(decision.Reason)) log.V(1).Info("Dropped audit event before recording", "reason", decision.Reason, "gvr", extractGVR(&event), "auditID", event.AuditID) - return acceptedFact{}, nil + return acceptedFact{} } eventRoute, routed := h.resolveEventRoute(ctx, route, &event) if !routed { - return acceptedFact{}, nil - } - - if h.config.FactRecorder != nil { - if err := h.config.FactRecorder.RecordFact(ctx, eventRoute, event); err != nil { - outcome.Record(ctx, &event, outcome.WriteError) - return acceptedFact{}, fmt.Errorf("record attribution fact %q: %w", event.AuditID, err) - } + return acceptedFact{} } - outcome.Record(ctx, &event, outcome.Queued) - h.firsts.factRecorded.Do(func() { - log.Info("Recorded first audit attribution fact", "auditID", event.AuditID, "verb", event.Verb) - }) - log.V(1).Info("Recorded audit attribution fact", - "gvr", extractGVR(&event), "verb", event.Verb, "auditID", event.AuditID, - "user", effectiveAuditUsername(event)) - return h.factForStream(ctx, eventRoute, event), nil + accepted := h.factForStream(ctx, eventRoute, event) + accepted.accepted = true + accepted.event = &event + return accepted } // factForStream reduces one accepted event to the fact its stream carries. It produces nothing when @@ -360,7 +409,7 @@ func (h *AuditHandler) factForStream(ctx context.Context, auditRoute string, eve if h.config.FactPublisher == nil { return acceptedFact{} } - fact, groupResource, ok := queue.AuthorFactFromEvent(ctx, event) + fact, groupResource, ok := queue.AuthorFactFromEvent(ctx, event, h.config.CollectionUIDCap) if !ok { return acceptedFact{} } @@ -371,24 +420,36 @@ func (h *AuditHandler) factForStream(ctx context.Context, auditRoute string, eve // server retries the delivery; a retried batch appends the same facts again under fresh stream IDs, // which is safe because a fact is keyed data rather than a position in a sequence — the duplicate // resolves to the same author and costs one entry's worth of retention. -func (h *AuditHandler) publishFactBatches(ctx context.Context, batches *factStreamBatches) error { +func (h *AuditHandler) publishFactBatches( + ctx context.Context, + batches *factStreamBatches, +) (map[queue.FactStreamKey]struct{}, error) { + appended := make(map[queue.FactStreamKey]struct{}, len(batches.order)) if h.config.FactPublisher == nil { - return nil + return appended, nil } for _, key := range batches.order { - if err := h.config.FactPublisher.PublishFacts(ctx, key, batches.facts[key]); err != nil { - return fmt.Errorf("publish attribution facts for %s: %w", key, err) + facts := batches.facts[key] + if err := h.config.FactPublisher.PublishFacts(ctx, key, facts); err != nil { + return appended, fmt.Errorf("publish attribution facts for %s: %w", key, err) } + queue.RecordFactsWritten(ctx, len(facts)) + appended[key] = struct{}{} } - return nil + return appended, nil } -// acceptedFact is one event's contribution to the request's appends, or the zero value when the -// event produced none. +// acceptedFact is one event's contribution to the request's appends. accepted and ok are NOT the +// same thing: an event that passed the accept gate but can never name an author — no user, no +// resolvable name and not a collection verb — is accepted (it gets a terminal outcome) and produces +// no fact (nothing is appended for it), because a waiter woken by a fact naming nobody has been +// woken for nothing. type acceptedFact struct { - key queue.FactStreamKey - fact queue.AuthorFact - ok bool + key queue.FactStreamKey + fact queue.AuthorFact + ok bool + accepted bool + event *auditv1.Event } // factStreamBatches groups one request's facts by the stream they belong to, keeping the order the diff --git a/internal/webhook/audit_handler_test.go b/internal/webhook/audit_handler_test.go index fcda981b..097753f1 100644 --- a/internal/webhook/audit_handler_test.go +++ b/internal/webhook/audit_handler_test.go @@ -20,6 +20,7 @@ import ( "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" + "github.com/ConfigButler/gitops-reverser/internal/queue" "github.com/ConfigButler/gitops-reverser/internal/telemetry" ) @@ -34,58 +35,65 @@ func TestMain(m *testing.M) { os.Exit(m.Run()) } -// fakeFactRecorder is an in-memory AuditFactRecorder. It appends every accepted -// event and can be told to fail with an injectable error. -type fakeFactRecorder struct { - mu sync.Mutex - err error - events []auditv1.Event - providers []string +// fakeFactSink is an in-memory AuditFactPublisher. It records every fact one request appended, +// with the audit route its stream was keyed on, and can be told to fail with an injectable error. +type fakeFactSink struct { + mu sync.Mutex + err error + facts []queue.AuthorFact + routes []string + // types records the group/resource each fact was filed under. It lives on the STREAM KEY now + // rather than in the fact, so this is where a test asserting "the right type was published" + // has to look. + types []string } -func (r *fakeFactRecorder) RecordFact(_ context.Context, providerName string, event auditv1.Event) error { +func (r *fakeFactSink) PublishFacts(_ context.Context, key queue.FactStreamKey, facts []queue.AuthorFact) error { r.mu.Lock() defer r.mu.Unlock() if r.err != nil { return r.err } - r.events = append(r.events, event) - r.providers = append(r.providers, providerName) + for _, fact := range facts { + r.facts = append(r.facts, fact) + r.routes = append(r.routes, key.AuditRoute) + r.types = append(r.types, key.GroupResource.String()) + } return nil } -// lastProvider returns the provider name threaded into the most recent RecordFact call. -func (r *fakeFactRecorder) lastProvider() string { +// lastProvider returns the audit route the most recently appended fact was filed under. +func (r *fakeFactSink) lastProvider() string { r.mu.Lock() defer r.mu.Unlock() - if len(r.providers) == 0 { + if len(r.routes) == 0 { return "" } - return r.providers[len(r.providers)-1] + return r.routes[len(r.routes)-1] } -// recordedProviders returns the provider name threaded into each RecordFact call, in order — the +// recordedProviders returns the audit route each appended fact was filed under, in order — the // fan-out a single annotation-routed batch produced. -func (r *fakeFactRecorder) recordedProviders() []string { +func (r *fakeFactSink) recordedProviders() []string { r.mu.Lock() defer r.mu.Unlock() - return append([]string(nil), r.providers...) + return append([]string(nil), r.routes...) } -func (r *fakeFactRecorder) auditIDs() []string { +func (r *fakeFactSink) auditIDs() []string { r.mu.Lock() defer r.mu.Unlock() - ids := make([]string, 0, len(r.events)) - for _, event := range r.events { - ids = append(ids, string(event.AuditID)) + ids := make([]string, 0, len(r.facts)) + for _, fact := range r.facts { + ids = append(ids, fact.AuditID) } return ids } -func (r *fakeFactRecorder) len() int { +func (r *fakeFactSink) len() int { r.mu.Lock() defer r.mu.Unlock() - return len(r.events) + return len(r.facts) } // eventListBody wraps zero or more event JSON fragments into an EventList body. @@ -140,8 +148,8 @@ const acceptedCreateEvent = `{"kind":"Event","level":"RequestResponse","auditID" // TestAuditHandler_NamedDefaultRouteThreadsItsProvider checks that /audit-webhook/default is an // ordinary named route: it records its facts under the "default" ClusterProvider name. func TestAuditHandler_NamedDefaultRouteThreadsItsProvider(t *testing.T) { - recorder := &fakeFactRecorder{} - handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactRecorder: recorder})) + recorder := &fakeFactSink{} + handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactPublisher: recorder})) require.NoError(t, err) body := `{"kind":"EventList","apiVersion":"audit.k8s.io/v1","items":[` + acceptedCreateEvent + `]}` @@ -159,8 +167,8 @@ func TestAuditHandler_NamedRouting(t *testing.T) { body := eventListBody(acceptedCreateEvent) t.Run("a named route records under its own name", func(t *testing.T) { - recorder := &fakeFactRecorder{} - handler, err := NewAuditHandler(AuditHandlerConfig{FactRecorder: recorder}) + recorder := &fakeFactSink{} + handler, err := NewAuditHandler(AuditHandlerConfig{FactPublisher: recorder}) require.NoError(t, err) w := serveBody(t, handler, http.MethodPost, "/audit-webhook/prod-eu-1", body) require.Equal(t, http.StatusOK, w.Code) @@ -168,8 +176,8 @@ func TestAuditHandler_NamedRouting(t *testing.T) { }) t.Run("a route no ClusterProvider declares is stored, not refused", func(t *testing.T) { - recorder := &fakeFactRecorder{} - handler, err := NewAuditHandler(AuditHandlerConfig{FactRecorder: recorder}) + recorder := &fakeFactSink{} + handler, err := NewAuditHandler(AuditHandlerConfig{FactPublisher: recorder}) require.NoError(t, err) w := serveBody(t, handler, http.MethodPost, "/audit-webhook/not-declared-yet", body) require.Equal(t, http.StatusOK, w.Code, @@ -179,8 +187,8 @@ func TestAuditHandler_NamedRouting(t *testing.T) { }) t.Run("default is an ordinary route", func(t *testing.T) { - recorder := &fakeFactRecorder{} - handler, err := NewAuditHandler(AuditHandlerConfig{FactRecorder: recorder}) + recorder := &fakeFactSink{} + handler, err := NewAuditHandler(AuditHandlerConfig{FactPublisher: recorder}) require.NoError(t, err) w := serveBody(t, handler, http.MethodPost, defaultRoute, body) require.Equal(t, http.StatusOK, w.Code) @@ -188,8 +196,8 @@ func TestAuditHandler_NamedRouting(t *testing.T) { }) t.Run("bare endpoint is 400 while no annotation key is configured", func(t *testing.T) { - recorder := &fakeFactRecorder{} - handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactRecorder: recorder})) + recorder := &fakeFactSink{} + handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactPublisher: recorder})) require.NoError(t, err) w := serveBody(t, handler, http.MethodPost, "/audit-webhook", body) assert.Equal(t, http.StatusBadRequest, w.Code, @@ -223,10 +231,10 @@ func annotatedEvent(auditID string, annotations map[string]string) string { // routes, and an event that names none is rejected by itself, never credited to a fallback, while // the rest of the batch still lands. A route no ClusterProvider has declared is NOT a rejection. func TestAuditHandler_AnnotationRouting(t *testing.T) { - newHandler := func(t *testing.T, recorder *fakeFactRecorder) *AuditHandler { + newHandler := func(t *testing.T, recorder *fakeFactSink) *AuditHandler { t.Helper() handler, err := NewAuditHandler(AuditHandlerConfig{ - FactRecorder: recorder, + FactPublisher: recorder, AuditRouteAnnotationKey: clusterAnnotation, }) require.NoError(t, err) @@ -234,7 +242,7 @@ func TestAuditHandler_AnnotationRouting(t *testing.T) { } t.Run("one batch fans out to several source clusters", func(t *testing.T) { - recorder := &fakeFactRecorder{} + recorder := &fakeFactSink{} handler := newHandler(t, recorder) w := serveBody(t, handler, http.MethodPost, "/audit-webhook", eventListBody( @@ -256,7 +264,7 @@ func TestAuditHandler_AnnotationRouting(t *testing.T) { {"a different key", map[string]string{"other.io/cluster": "prod-eu-1"}}, } { t.Run(tt.name, func(t *testing.T) { - recorder := &fakeFactRecorder{} + recorder := &fakeFactSink{} handler := newHandler(t, recorder) w := serveBody(t, handler, http.MethodPost, "/audit-webhook", eventListBody( @@ -273,7 +281,7 @@ func TestAuditHandler_AnnotationRouting(t *testing.T) { }) t.Run("a route no ClusterProvider declares is stored like any other", func(t *testing.T) { - recorder := &fakeFactRecorder{} + recorder := &fakeFactSink{} handler := newHandler(t, recorder) w := serveBody(t, handler, http.MethodPost, "/audit-webhook", eventListBody( @@ -285,7 +293,7 @@ func TestAuditHandler_AnnotationRouting(t *testing.T) { }) t.Run("named routes ignore the annotation", func(t *testing.T) { - recorder := &fakeFactRecorder{} + recorder := &fakeFactSink{} handler := newHandler(t, recorder) w := serveBody(t, handler, http.MethodPost, "/audit-webhook/prod-us-1", eventListBody( @@ -399,8 +407,8 @@ func TestAuditHandler_DecodeErrors(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - recorder := &fakeFactRecorder{} - handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactRecorder: recorder})) + recorder := &fakeFactSink{} + handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactPublisher: recorder})) require.NoError(t, err) w := serveBody(t, handler, http.MethodPost, defaultRoute, tt.body) @@ -411,8 +419,8 @@ func TestAuditHandler_DecodeErrors(t *testing.T) { } func TestAuditHandler_RejectsOversizedBody(t *testing.T) { - recorder := &fakeFactRecorder{} - handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{MaxRequestBodyBytes: 32, FactRecorder: recorder})) + recorder := &fakeFactSink{} + handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{MaxRequestBodyBytes: 32, FactPublisher: recorder})) require.NoError(t, err) w := serveBody(t, handler, http.MethodPost, defaultRoute, eventListBody(acceptedCreateEvent)) @@ -422,8 +430,8 @@ func TestAuditHandler_RejectsOversizedBody(t *testing.T) { } func TestAuditHandler_EmptyEventListRecordsNothing(t *testing.T) { - recorder := &fakeFactRecorder{} - handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactRecorder: recorder})) + recorder := &fakeFactSink{} + handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactPublisher: recorder})) require.NoError(t, err) w := serveBody(t, handler, http.MethodPost, defaultRoute, eventListBody()) @@ -434,8 +442,8 @@ func TestAuditHandler_EmptyEventListRecordsNothing(t *testing.T) { // TestAuditHandler_AcceptedEventRecordsFact is the happy path: a canonical // mutating event reaches the FactRecorder and the request returns 200. func TestAuditHandler_AcceptedEventRecordsFact(t *testing.T) { - recorder := &fakeFactRecorder{} - handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactRecorder: recorder})) + recorder := &fakeFactSink{} + handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactPublisher: recorder})) require.NoError(t, err) w := serveBody(t, handler, http.MethodPost, defaultRoute, eventListBody(acceptedCreateEvent)) @@ -456,8 +464,8 @@ func TestAuditHandler_NilRecorderAcceptsWithoutRecording(t *testing.T) { // TestAuditHandler_RecordsEveryAcceptedEventInBatch confirms the handler walks // the whole list, recording each accepted event in order. func TestAuditHandler_RecordsEveryAcceptedEventInBatch(t *testing.T) { - recorder := &fakeFactRecorder{} - handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactRecorder: recorder})) + recorder := &fakeFactSink{} + handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactPublisher: recorder})) require.NoError(t, err) second := `{"kind":"Event","auditID":"update-1","stage":"ResponseComplete","verb":"update",` + @@ -474,8 +482,8 @@ func TestAuditHandler_RecordsEveryAcceptedEventInBatch(t *testing.T) { // TestAuditHandler_RecordFactErrorFailsRequest pins the retry contract: a // fact-store failure surfaces as 500 so the API server redelivers. func TestAuditHandler_RecordFactErrorFailsRequest(t *testing.T) { - recorder := &fakeFactRecorder{err: errors.New("fact store down")} - handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactRecorder: recorder})) + recorder := &fakeFactSink{err: errors.New("fact store down")} + handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactPublisher: recorder})) require.NoError(t, err) w := serveBody(t, handler, http.MethodPost, defaultRoute, eventListBody(acceptedCreateEvent)) @@ -563,8 +571,8 @@ func TestAuditHandler_RejectedEventsAreDropped(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - recorder := &fakeFactRecorder{} - handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactRecorder: recorder})) + recorder := &fakeFactSink{} + handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactPublisher: recorder})) require.NoError(t, err) w := serveBody(t, handler, http.MethodPost, defaultRoute, eventListBody(tt.event)) @@ -586,7 +594,9 @@ func TestAuditHandler_AcceptedEdgeCases(t *testing.T) { { name: "scale subresource forwards", event: `{"kind":"Event","auditID":"scale-1","stage":"ResponseComplete","verb":"patch",` + - `"objectRef":{"resource":"deployments","apiGroup":"apps","apiVersion":"apps/v1","subresource":"scale"},` + + `"user":{"username":"alice"},` + + `"objectRef":{"resource":"deployments","apiGroup":"apps","apiVersion":"apps/v1",` + + `"name":"web","subresource":"scale"},` + `"responseStatus":{"code":200},` + `"responseObject":{"kind":"Scale","metadata":{"resourceVersion":"5"}}}`, auditID: "scale-1", @@ -594,6 +604,7 @@ func TestAuditHandler_AcceptedEdgeCases(t *testing.T) { { name: "bodyless delete is accepted", event: `{"kind":"Event","auditID":"delete-1","stage":"ResponseComplete","verb":"delete",` + + `"user":{"username":"alice"},` + `"objectRef":{"resource":"configmaps","apiVersion":"v1","name":"cm"},` + `"responseStatus":{"code":200}}`, auditID: "delete-1", @@ -601,7 +612,8 @@ func TestAuditHandler_AcceptedEdgeCases(t *testing.T) { { name: "missing response status is accepted", event: `{"kind":"Event","auditID":"nostatus-1","stage":"ResponseComplete","verb":"create",` + - `"objectRef":{"resource":"configmaps","apiVersion":"v1"},` + + `"user":{"username":"alice"},` + + `"objectRef":{"resource":"configmaps","apiVersion":"v1","name":"cm"},` + `"responseObject":{"metadata":{"resourceVersion":"5"}}}`, auditID: "nostatus-1", }, @@ -609,8 +621,8 @@ func TestAuditHandler_AcceptedEdgeCases(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - recorder := &fakeFactRecorder{} - handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactRecorder: recorder})) + recorder := &fakeFactSink{} + handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactPublisher: recorder})) require.NoError(t, err) w := serveBody(t, handler, http.MethodPost, defaultRoute, eventListBody(tt.event)) @@ -620,12 +632,13 @@ func TestAuditHandler_AcceptedEdgeCases(t *testing.T) { } } -// TestAuditHandler_BatchStopsOnFirstRecordError confirms a fact-store failure on -// an earlier event short-circuits the batch: later events are not recorded and -// the whole request is 500. +// TestAuditHandler_BatchStopsOnFirstRecordError confirms a transport failure fails the whole +// request: nothing from the batch reaches the fact log and the API server gets a 500, so it +// retries the delivery. A retried batch appends the same facts under fresh stream IDs, which is +// safe because a fact is keyed data rather than a position in a sequence. func TestAuditHandler_BatchStopsOnFirstRecordError(t *testing.T) { - recorder := &fakeFactRecorder{err: errors.New("fact store down")} - handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactRecorder: recorder})) + recorder := &fakeFactSink{err: errors.New("fact store down")} + handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactPublisher: recorder})) require.NoError(t, err) second := `{"kind":"Event","auditID":"update-1","stage":"ResponseComplete","verb":"update",` + @@ -639,25 +652,23 @@ func TestAuditHandler_BatchStopsOnFirstRecordError(t *testing.T) { // TestAuditHandler_ForwardsRealScaleSubresourceRecording drives the captured // `kubectl scale deployment` recording through the full path and asserts the -// deployments/scale event reaches the FactRecorder with verb/resource/subresource +// deployments/scale event reaches the fact log with verb/resource/subresource // intact rather than being dropped. func TestAuditHandler_ForwardsRealScaleSubresourceRecording(t *testing.T) { recording, err := os.ReadFile("testdata/audit-events/deployment-scale-subresource.json") require.NoError(t, err, "the captured scale recording must be readable") - recorder := &fakeFactRecorder{} - handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactRecorder: recorder})) + recorder := &fakeFactSink{} + handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactPublisher: recorder})) require.NoError(t, err) w := serveBody(t, handler, http.MethodPost, defaultRoute, eventListBody(string(recording))) require.Equal(t, http.StatusOK, w.Code) require.Equal(t, 1, recorder.len(), "the real deployments/scale recording must be recorded") - got := recorder.events[0] - require.NotNil(t, got.ObjectRef) - assert.Equal(t, "deployments", got.ObjectRef.Resource) - assert.Equal(t, "scale", got.ObjectRef.Subresource) - assert.Equal(t, "patch", got.Verb) + // The type is the stream's identity, so it is asserted on the key rather than on the fact. + assert.Equal(t, "deployments.apps", recorder.types[0], "the scale target keeps the deployment's type") + assert.Equal(t, "patch", recorder.facts[0].Verb, "a /scale patch is published as the patch it is") } // TestAuditHandler_FixtureDryRunAndUnchangedRVDropped drives the real captured @@ -673,8 +684,8 @@ func TestAuditHandler_FixtureDryRunAndUnchangedRVDropped(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - recorder := &fakeFactRecorder{} - handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactRecorder: recorder})) + recorder := &fakeFactSink{} + handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactPublisher: recorder})) require.NoError(t, err) w := serveBody(t, handler, http.MethodPost, defaultRoute, eventListFixtureBody(t, tt.fixture)) @@ -707,8 +718,8 @@ func TestAuditHandler_FixturePersistedAndCreateRecorded(t *testing.T) { for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { - recorder := &fakeFactRecorder{} - handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactRecorder: recorder})) + recorder := &fakeFactSink{} + handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactPublisher: recorder})) require.NoError(t, err) w := serveBody(t, handler, http.MethodPost, defaultRoute, eventListFixtureBody(t, tt.fixture)) diff --git a/internal/webhook/audit_identity_test.go b/internal/webhook/audit_identity_test.go index cc65bd50..8ab3f669 100644 --- a/internal/webhook/audit_identity_test.go +++ b/internal/webhook/audit_identity_test.go @@ -105,8 +105,8 @@ func TestExtractGVR_CoreGroupRendersLeadingSlash(t *testing.T) { // for an impersonated mutation, so the identity rule above is proven end-to-end rather than only at // the helper. func TestAuditHandler_ImpersonatedEventIsRecordedUnderTheImpersonatedUser(t *testing.T) { - recorder := &fakeFactRecorder{} - handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactRecorder: recorder})) + recorder := &fakeFactSink{} + handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactPublisher: recorder})) require.NoError(t, err) const impersonated = `{"kind":"Event","level":"RequestResponse","auditID":"imp-1",` + @@ -120,6 +120,6 @@ func TestAuditHandler_ImpersonatedEventIsRecordedUnderTheImpersonatedUser(t *tes w := serveBody(t, handler, http.MethodPost, defaultRoute, eventListBody(impersonated)) require.Equal(t, http.StatusOK, w.Code) require.Equal(t, 1, recorder.len()) - assert.Equal(t, "alice", effectiveAuditUsername(recorder.events[0]), + assert.Equal(t, "alice", recorder.facts[0].Author, "the impersonated actor is the author, not the account permitted to impersonate") } diff --git a/internal/webhook/audit_metrics_test.go b/internal/webhook/audit_metrics_test.go index 9f8abf62..28a9b59e 100644 --- a/internal/webhook/audit_metrics_test.go +++ b/internal/webhook/audit_metrics_test.go @@ -87,11 +87,11 @@ func TestServeHTTP_EventListIngressMetrics(t *testing.T) { reader, err := telemetry.InitTestExporter() require.NoError(t, err) - recorder := &fakeFactRecorder{} + recorder := &fakeFactSink{} if tt.recorderErr { recorder.err = errAuditTest } - handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactRecorder: recorder})) + handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactPublisher: recorder})) require.NoError(t, err) w := serveBody(t, handler, http.MethodPost, defaultRoute, tt.body) @@ -124,8 +124,8 @@ func TestServeHTTP_AcceptedEventQueuedOutcome(t *testing.T) { reader, err := telemetry.InitTestExporter() require.NoError(t, err) - recorder := &fakeFactRecorder{} - handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactRecorder: recorder})) + recorder := &fakeFactSink{} + handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactPublisher: recorder})) require.NoError(t, err) w := serveBody(t, handler, http.MethodPost, defaultRoute, eventListBody(acceptedCreateEvent)) @@ -157,9 +157,9 @@ func TestServeHTTP_UnroutableEventOutcomes(t *testing.T) { reader, err := telemetry.InitTestExporter() require.NoError(t, err) - recorder := &fakeFactRecorder{} + recorder := &fakeFactSink{} handler, err := NewAuditHandler(AuditHandlerConfig{ - FactRecorder: recorder, + FactPublisher: recorder, AuditRouteAnnotationKey: clusterAnnotation, }) require.NoError(t, err) @@ -187,8 +187,8 @@ func TestServeHTTP_NonScaleSubresourceDropped(t *testing.T) { reader, err := telemetry.InitTestExporter() require.NoError(t, err) - recorder := &fakeFactRecorder{} - handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactRecorder: recorder})) + recorder := &fakeFactSink{} + handler, err := NewAuditHandler(routedConfig(AuditHandlerConfig{FactPublisher: recorder})) require.NoError(t, err) w := serveBody(t, handler, http.MethodPost, defaultRoute, eventListBody(subresourceExecEvent)) diff --git a/test/e2e/aggregated_deletecollection_e2e_test.go b/test/e2e/aggregated_deletecollection_e2e_test.go new file mode 100644 index 00000000..001811f6 --- /dev/null +++ b/test/e2e/aggregated_deletecollection_e2e_test.go @@ -0,0 +1,254 @@ +// SPDX-License-Identifier: Apache-2.0 + +package e2e + +import ( + "context" + "fmt" + "os" + "path" + "path/filepath" + "time" + + . "github.com/onsi/ginkgo/v2" + . "github.com/onsi/gomega" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/client-go/dynamic" + "k8s.io/client-go/rest" + "k8s.io/client-go/tools/clientcmd" +) + +// Aggregated deletecollection attribution is the case the deleted response-body expander +// gave up on entirely, driven against a real aggregated API server rather than a fixture. +// +// The kube-apiserver PROXIES a request for an aggregated resource to the extension server. It +// audits the request it proxied, but it never decodes the response it streamed back, so +// responseObject is empty — row 15 of the lab corpus established exactly that for an +// aggregated-API create (test/mutationlab/README.md), and a collection delete travels the same +// proxy path. The expander needed that body to reconstruct one fact per object, so for wardle it +// produced nothing and every removal shipped committer-authored. +// +// The collection fact does not need it. The audit event still carries what the actor asked for — +// the type, the namespace, the selector on the request URI, and the stage timestamp — and a removal +// joins that by SCOPE. This spec asserts the outcome that follows: every removal commit is authored +// by the actor who ran the collection delete. +// +// It deliberately asserts the AUTHOR rather than the metric tier. Which tier fires is a fact about +// the API server's proxy behaviour, not about this operator: if some future apiserver did return a +// body, the join would silently upgrade to uid membership and this spec should still pass. The tier +// that actually fired is reported, not asserted, so a change in that behaviour is visible without +// being a failure. +// +// Not Serial: the wardle APIService is installed once at cluster setup and only read here; the +// collection delete is scoped by a per-run label selector, so concurrent specs cannot be caught by +// it. See docs/spec/e2e-serial-registry.md. +var _ = Describe("Aggregated API deletecollection attribution", Label("aggregated-api"), Ordered, func() { + var ( + testNs string + repo *RepoArtifacts + providerName string + targetName string + watchRuleName string + basePath string + alice dynamic.Interface + ) + + BeforeAll(func() { + if configuredAuthorModeEnabled() { + Skip("watch-first configured-author mode has no audit facts for delete attribution") + } + + testNs = testNamespaceFor("agg-dc") + providerName = "agg-dc-provider" + targetName = "agg-dc-target" + watchRuleName = "agg-dc-watchrule" + basePath = "e2e/aggregated-deletecollection" + + _, _ = kubectlRun("create", "namespace", testNs) + + repo = SetupRepo( + resolveE2EContext(), + testNs, + fmt.Sprintf("e2e-agg-dc-%d", GinkgoRandomSeed()), + ) + _, err := kubectlRunInNamespace(testNs, "apply", "-f", repo.SecretsYAML) + Expect(err).NotTo(HaveOccurred(), "failed to apply git secrets to the aggregated-dc namespace") + applySOPSAgeKeyToNamespace(testNs) + + By("setting up GitProvider (0s commit window), GitTarget and a flunder WatchRule") + createReadyGitProvider(providerName, testNs, repo.GitSecretHTTP, repo.RepoURLHTTP) + createValidatedGitTarget(targetName, testNs, providerName, basePath) + Expect(applyFromTemplate( + "test/e2e/templates/aggregated-api/watchrule-flunder.tmpl", + struct { + Name string + Namespace string + GitTargetName string + }{Name: watchRuleName, Namespace: testNs, GitTargetName: targetName}, + testNs, + )).To(Succeed()) + verifyResourceStatus("watchrule", watchRuleName, testNs, "True", "Succeeded", "") + waitForStreamsRunning(targetName, testNs) + + By("building an impersonated dynamic client for the actor") + alice, err = impersonatedDynamicClient("oidc-alice", "Alice Liddell", "alice@configbutler.ai") + Expect(err).NotTo(HaveOccurred()) + }) + + AfterAll(func() { + cleanupPipeline(testNs, providerName, targetName, watchRuleName) + cleanupNamespace(testNs) + }) + + It("attributes a body-less aggregated collection delete to the actor", func() { + label := fmt.Sprintf("aggdc-%d", GinkgoRandomSeed()) + doomedA := label + "-doomed-a" + doomedB := label + "-doomed-b" + survivor := label + "-survivor" + + flunderPath := func(name string) string { + return path.Join(basePath, fmt.Sprintf("%s/wardle.example.com/flunders/%s.yaml", testNs, name)) + } + + By("creating two flunders the collection will cover and one it will not") + for name, tier := range map[string]string{doomedA: "doomed", doomedB: "doomed", survivor: "keep"} { + Expect(createLabeledFlunder(alice, testNs, name, label, tier)).To(Succeed()) + } + for _, name := range []string{doomedA, doomedB, survivor} { + waitForFilePresent(repo, flunderPath(name)) + } + + before := collectionResolutionCounts() + + By("deleting only the doomed subset as the actor, through the aggregation layer") + Expect(deleteFlunderCollection(alice, testNs, "aggdc="+label+",tier=doomed")).To(Succeed()) + + By("asserting both removals are authored by the actor, not the committer") + waitForFileDeletedByActor(repo, flunderPath(doomedA)) + waitForFileDeletedByActor(repo, flunderPath(doomedB)) + + By("asserting the flunder outside the selector survives untouched") + Consistently(func(g Gomega) { + pullLatestRepoState(g, repo.CheckoutDir) + _, statErr := os.Stat(filepath.Join(repo.CheckoutDir, flunderPath(survivor))) + g.Expect(statErr).NotTo(HaveOccurred(), + "a flunder the selector did not match must not be removed") + }, 8*time.Second, 2*time.Second).Should(Succeed()) + + reportCollectionTier(before) + }) +}) + +// collectionResolutionCounts snapshots the two collection tiers so the spec can report which one +// the aggregated collection delete actually took. +func collectionResolutionCounts() map[string]float64 { + // Without this the queries below fail on a nil client and the tier goes unreported, which is a + // silent hole rather than a failure: the spec asserts the author, so it passes either way. + ensurePrometheusClient() + counts := map[string]float64{} + for _, tier := range []string{"collection_uid", "collection_scope"} { + n, err := queryPrometheus(fmt.Sprintf( + `sum(max_over_time(gitopsreverser_attribution_resolutions_total{tier=%q}[2h])) or vector(0)`, tier)) + if err != nil { + return map[string]float64{} + } + counts[tier] = n + } + return counts +} + +// reportCollectionTier says which collection tier resolved this spec's removals. It reports rather +// than asserts: whether the API server sends a response body for a proxied collection delete is a +// fact about the API server, and the join is correct either way. +func reportCollectionTier(before map[string]float64) { + if len(before) == 0 { + return + } + after := collectionResolutionCounts() + uid := after["collection_uid"] - before["collection_uid"] + scope := after["collection_scope"] - before["collection_scope"] + switch { + case scope > 0 && uid == 0: + _, _ = fmt.Fprintf(GinkgoWriter, + "\nℹ️ aggregated deletecollection resolved by SCOPE (+%.0f collection_scope): the proxied "+ + "request carried no response body, which is exactly the case the deleted expander "+ + "produced nothing for.\n", scope) + case uid > 0: + _, _ = fmt.Fprintf(GinkgoWriter, + "\nℹ️ aggregated deletecollection resolved by UID membership (+%.0f collection_uid, "+ + "+%.0f collection_scope): this API server DID return the deleted set for a proxied "+ + "collection delete.\n", uid, scope) + default: + _, _ = fmt.Fprintf(GinkgoWriter, + "\nℹ️ aggregated deletecollection resolved through a stronger per-object tier: neither "+ + "collection tier moved, so each removal found its own fact first.\n") + } +} + +// flunderGVR is the aggregated type this spec mirrors. +var flunderGVR = schema.GroupVersionResource{ + Group: "wardle.example.com", + Version: "v1alpha1", + Resource: "flunders", +} + +// impersonatedDynamicClient builds a dynamic client that impersonates asUser with OIDC +// display-name/email claims, so writes it makes are attributed to " " in Git. +// It is the aggregated-type sibling of impersonatedConfigMapClient, which is typed and therefore +// cannot reach wardle. +func impersonatedDynamicClient(asUser, displayName, email string) (dynamic.Interface, error) { + loadingRules := clientcmd.NewDefaultClientConfigLoadingRules() + overrides := &clientcmd.ConfigOverrides{} + if ctx := kubectlContext(); ctx != "" { + overrides.CurrentContext = ctx + } + config, err := clientcmd.NewNonInteractiveDeferredLoadingClientConfig(loadingRules, overrides).ClientConfig() + if err != nil { + return nil, fmt.Errorf("load kubeconfig: %w", err) + } + config.Impersonate = rest.ImpersonationConfig{ + UserName: asUser, + Groups: []string{"system:masters"}, + Extra: map[string][]string{ + "configbutler.ai/claims/display-name": {displayName}, + "configbutler.ai/claims/email": {email}, + }, + } + return dynamic.NewForConfig(config) +} + +// createLabeledFlunder creates one flunder carrying the per-run aggdc label and a tier label the +// collection delete selects on. +func createLabeledFlunder(client dynamic.Interface, ns, name, aggdc, tier string) error { + flunder := &unstructured.Unstructured{Object: map[string]any{ + "apiVersion": "wardle.example.com/v1alpha1", + "kind": "Flunder", + "metadata": map[string]any{ + "name": name, + "namespace": ns, + "labels": map[string]any{"aggdc": aggdc, "tier": tier}, + }, + "spec": map[string]any{"reference": "aggregated-deletecollection"}, + }} + _, err := client.Resource(flunderGVR).Namespace(ns). + Create(context.Background(), flunder, metav1.CreateOptions{}) + if err != nil { + return fmt.Errorf("create flunder %s: %w", name, err) + } + return nil +} + +// deleteFlunderCollection issues the name-less collection delete this spec is about. +func deleteFlunderCollection(client dynamic.Interface, ns, labelSelector string) error { + err := client.Resource(flunderGVR).Namespace(ns).DeleteCollection( + context.Background(), + metav1.DeleteOptions{}, + metav1.ListOptions{LabelSelector: labelSelector}, + ) + if err != nil { + return fmt.Errorf("deletecollection flunders (%s): %w", labelSelector, err) + } + return nil +} diff --git a/test/e2e/author_mode_test.go b/test/e2e/author_mode_test.go index 39e091a5..89ddf15e 100644 --- a/test/e2e/author_mode_test.go +++ b/test/e2e/author_mode_test.go @@ -40,6 +40,16 @@ func TestConfiguredAuthorModeFromArgs_AllScenarios(t *testing.T) { {"upper false", `["--author-attribution=FALSE","--redis-addr="]`, true}, {"numeric true is not an opt-out", `["--author-attribution=1"]`, false}, {"capitalised true is not an opt-out", `["--author-attribution=True"]`, false}, + + // Attribution needs a fact TRANSPORT, which stopped meaning Redis. The in-memory transport + // runs it with no Redis at all, so reading an empty --redis-addr as configured-author would + // skip every attribution spec and report a green run that asserted nothing about the mode + // it was supposed to be exercising. + {"memory transport needs no redis", `["--redis-addr=","--author-attribution-transport=memory"]`, false}, + {"redis transport with no redis is still configured-author", + `["--redis-addr=","--author-attribution-transport=redis"]`, true}, + {"attribution off beats any transport", + `["--author-attribution=false","--author-attribution-transport=memory"]`, true}, } { t.Run(tc.name, func(t *testing.T) { if got := configuredAuthorModeFromArgs(tc.args); got != tc.want { diff --git a/test/e2e/commit_request_e2e_test.go b/test/e2e/commit_request_e2e_test.go index 32833aa0..96a3a977 100644 --- a/test/e2e/commit_request_e2e_test.go +++ b/test/e2e/commit_request_e2e_test.go @@ -106,17 +106,11 @@ var _ = Describe("Commit Request", Label("commit-request", "audit-consumer"), Or By("creating a CommitRequest to finalize the open window now") applyCommitRequest(testNs, commitRequestName, gitTargetName, message) - By("waiting for the CommitRequest to become Ready (committed and pushed)") + By("waiting for the CommitRequest to commit and push") var reportedSHA string Eventually(func(g Gomega) { - g.Expect(commitRequestCondition(g, testNs, commitRequestName, "Ready")).To(Equal("True"), - "CommitRequest should finalize the window and become Ready\n%s", + reportedSHA = expectCommitRequestCommitted(g, testNs, commitRequestName, recentCommitDiagnostics(repo.CheckoutDir, basePath)) - g.Expect(commitRequestCondition(g, testNs, commitRequestName, "Pushed")).To(Equal("True"), - "a committed CommitRequest must report Pushed=True") - - reportedSHA = commitRequestField(g, testNs, commitRequestName, "{.status.sha}") - g.Expect(reportedSHA).NotTo(BeEmpty(), "status.sha should be populated") branch := commitRequestField(g, testNs, commitRequestName, "{.status.branch}") g.Expect(branch).To(Equal("main")) @@ -188,8 +182,8 @@ var _ = Describe("Commit Request", Label("commit-request", "audit-consumer"), Or By("creating a CommitRequest and confirming the branch then advances with the held edit") applyCommitRequest(testNs, commitRequestName, gitTargetName, message) Eventually(func(g Gomega) { - g.Expect(commitRequestCondition(g, testNs, commitRequestName, "Ready")).To(Equal("True"), - "CommitRequest should finalize the open window") + expectCommitRequestCommitted(g, testNs, commitRequestName, + recentCommitDiagnostics(repo.CheckoutDir, basePath)) g.Expect(remoteBranchHead(g, repo.CheckoutDir)).NotTo(Equal(baseSHA), "finalizing must advance main past the previously-established HEAD") }, 2*time.Minute, 3*time.Second).Should(Succeed()) @@ -231,15 +225,10 @@ var _ = Describe("Commit Request", Label("commit-request", "audit-consumer"), Or By("creating a CommitRequest with metadata.generateName") generatedName := applyCommitRequestWithGenerateName(testNs, commitRequestPrefix, gitTargetName, message) - By("waiting for the generated-name CommitRequest to become Ready") - var reportedSHA string + By("waiting for the generated-name CommitRequest to commit") Eventually(func(g Gomega) { - g.Expect(commitRequestCondition(g, testNs, generatedName, "Ready")).To(Equal("True"), - "a CommitRequest created via generateName must become Ready\n%s", + expectCommitRequestCommitted(g, testNs, generatedName, recentCommitDiagnostics(repo.CheckoutDir, basePath)) - - reportedSHA = commitRequestField(g, testNs, generatedName, "{.status.sha}") - g.Expect(reportedSHA).NotTo(BeEmpty(), "status.sha should be populated") }, 2*time.Minute, 3*time.Second).Should(Succeed()) By("verifying the commit landed in Git with the explicit message") @@ -362,16 +351,12 @@ var _ = Describe("Commit Request Bundle (UC2)", Label("commit-request", "audit-c _, err := kubectlRunWithStdin(testNs, bundle.String(), "apply", "-f", "-") Expect(err).NotTo(HaveOccurred(), "failed to apply the CommitRequest+Deployments bundle") - By("waiting for the bundle's CommitRequest to become Ready") + By("waiting for the bundle's CommitRequest to commit the collected window") var reportedSHA string Eventually(func(g Gomega) { - g.Expect(commitRequestCondition(g, testNs, commitRequestName, "Ready")).To(Equal("True"), - "the bundle's CommitRequest should finalize the collected window and become Ready\n%s", + reportedSHA = expectCommitRequestCommitted(g, testNs, commitRequestName, recentCommitDiagnostics(repo.CheckoutDir, basePath)) - reportedSHA = commitRequestField(g, testNs, commitRequestName, "{.status.sha}") - g.Expect(reportedSHA).NotTo(BeEmpty(), "status.sha should be populated") - branch := commitRequestField(g, testNs, commitRequestName, "{.status.branch}") g.Expect(branch).To(Equal("main")) }, 2*time.Minute, 3*time.Second).Should(Succeed()) @@ -519,3 +504,79 @@ func commitRequestCondition(g Gomega, namespace, name, conditionType string) str return commitRequestField(g, namespace, name, fmt.Sprintf(`{.status.conditions[?(@.type=="%s")].status}`, conditionType)) } + +// commitRequestReasonCommitted is the one Ready reason that means a commit actually happened. +// It mirrors crReasonCommitted in internal/controller/commitrequest_finalize.go. +const commitRequestReasonCommitted = "Committed" + +// commitRequestOutcome is the terminal shape of a CommitRequest: not just whether it finished, +// but WHICH ending it reached. +// +// Ready=True alone cannot answer that, and reading it as "it committed" is how this suite went +// blind. A benign rejection — no window collected in the grace, a foreign author's window, a +// change already present on the remote — is deliberately Ready=True with Pushed=False and an +// empty status.sha, so that kstatus reads Current rather than Failed (see rejectCommitRequest). +// A spec asserting Ready=True therefore PASSES on a request that committed nothing, then fails +// seconds later on the empty sha with ": not to be empty" and no reason attached. +type commitRequestOutcome struct { + Ready string + Reason string + Message string + Stalled string + Pushed string + SHA string + Branch string +} + +// String renders the outcome as the diagnostic the bare-sha assertion never produced. +func (o commitRequestOutcome) String() string { + return fmt.Sprintf( + "CommitRequest status: Ready=%s (reason=%q) Stalled=%s Pushed=%s sha=%q branch=%q\n message: %s", + o.Ready, o.Reason, o.Stalled, o.Pushed, o.SHA, o.Branch, o.Message) +} + +// isTerminal reports whether the controller has stopped working on this request: Ready=True +// (committed or benignly rejected) or Stalled=True (failed). It mirrors commitRequestIsTerminal +// in the controller, and it is what makes an early give-up safe. +func (o commitRequestOutcome) isTerminal() bool { + return o.Ready == "True" || o.Stalled == "True" +} + +// readCommitRequestOutcome reads the whole terminal picture in one API call, so every field in a +// failure report is from the same observation rather than five reads a poll apart. +func readCommitRequestOutcome(g Gomega, namespace, name string) commitRequestOutcome { + const readyPath = `{.status.conditions[?(@.type=="Ready")]` + return commitRequestOutcome{ + Ready: commitRequestField(g, namespace, name, readyPath+`.status}`), + Reason: commitRequestField(g, namespace, name, readyPath+`.reason}`), + Message: commitRequestField(g, namespace, name, readyPath+`.message}`), + Stalled: commitRequestCondition(g, namespace, name, "Stalled"), + Pushed: commitRequestCondition(g, namespace, name, "Pushed"), + SHA: commitRequestField(g, namespace, name, "{.status.sha}"), + Branch: commitRequestField(g, namespace, name, "{.status.branch}"), + } +} + +// expectCommitRequestCommitted asserts the request reached the COMMITTED ending, and returns the +// SHA it reported. +// +// It gives up early on any other terminal ending rather than polling to the timeout. A terminal +// outcome is final — the controller will not revisit it — so continuing to re-read it cannot +// change the result; it only delays the report and buries the reason. Failing at the moment the +// ending is known means the message names it: "NoWindowInGrace" instead of an empty string. +func expectCommitRequestCommitted(g Gomega, namespace, name, diagnostics string) string { + outcome := readCommitRequestOutcome(g, namespace, name) + if outcome.isTerminal() && outcome.Reason != commitRequestReasonCommitted { + StopTrying(fmt.Sprintf( + "the CommitRequest reached a terminal outcome that is not a commit, so waiting longer "+ + "cannot help.\n%s\n%s", outcome, diagnostics)).Now() + } + + g.Expect(outcome.Ready).To(Equal("True"), "CommitRequest is not Ready yet\n%s\n%s", outcome, diagnostics) + g.Expect(outcome.Reason).To(Equal(commitRequestReasonCommitted), + "Ready=True must mean a commit, not a benign rejection\n%s\n%s", outcome, diagnostics) + g.Expect(outcome.Pushed).To(Equal("True"), + "a committed CommitRequest must report Pushed=True\n%s", outcome) + g.Expect(outcome.SHA).NotTo(BeEmpty(), "a committed CommitRequest must report status.sha\n%s", outcome) + return outcome.SHA +} diff --git a/test/e2e/deletecollection_intent_e2e_test.go b/test/e2e/deletecollection_intent_e2e_test.go index a3a2eecb..2bee5301 100644 --- a/test/e2e/deletecollection_intent_e2e_test.go +++ b/test/e2e/deletecollection_intent_e2e_test.go @@ -203,6 +203,40 @@ var _ = Describe("DeleteCollection intent & attribution", Label("manager"), Orde Expect(removeConfigMapFinalizers(cleanupBot, testNs, name)).To(Succeed()) }) + // The other specs create and delete as the SAME actor, so a removal credited to the last writer + // instead of the deleter produces the same name and passes. This one separates them: the cleanup + // bot writes the objects, alice deletes the collection, and only the deleter may be credited. + // Getting that wrong is not a cosmetic error — it names an innocent person as the author of a + // deletion they did not perform. + It("credits the collection deleter, not whoever last edited the objects", func() { + label := fmt.Sprintf("dctest-deleter-%d", GinkgoRandomSeed()) + first := label + "-a" + second := label + "-b" + + By("creating the configmaps as alice, then having a DIFFERENT identity edit them last") + Expect(createLabeledConfigMap(alice, testNs, first, label, nil, "tier", "doomed")).To(Succeed()) + Expect(createLabeledConfigMap(alice, testNs, second, label, nil, "tier", "doomed")).To(Succeed()) + for _, n := range []string{first, second} { + waitForFilePresent(repo, configMapRepoPath(testNs, n)) + Expect(annotateConfigMap(cleanupBot, testNs, n, "touched-by", "cleanup-bot")).To(Succeed()) + } + + // Waiting for the edit to be COMMITTED, rather than merely requested, is what keeps this + // spec honest. The premise is that the cleanup bot is the last writer when the collection is + // deleted; if the delete raced ahead of the edit, alice would still be the last writer and + // the assertion below would pass without ever exercising the case it exists for. + By("waiting until the cleanup bot is demonstrably the last writer of each object") + waitForFileAuthoredBy(repo, configMapRepoPath(testNs, first), dcIntentEditorAuthor) + waitForFileAuthoredBy(repo, configMapRepoPath(testNs, second), dcIntentEditorAuthor) + + By("deleting the collection as alice") + Expect(deleteConfigMapCollection(alice, testNs, "dctest="+label)).To(Succeed()) + + By("asserting the removals are authored by alice, the deleter — never by the last editor") + waitForFileDeletedByActor(repo, configMapRepoPath(testNs, first)) + waitForFileDeletedByActor(repo, configMapRepoPath(testNs, second)) + }) + It("scopes a label-selector collection delete to matching objects and leaves siblings", func() { label := fmt.Sprintf("dctest-selector-%d", GinkgoRandomSeed()) matchA := label + "-match-a" @@ -239,6 +273,9 @@ var _ = Describe("DeleteCollection intent & attribution", Label("manager"), Orde const ( dcIntentBasePath = "e2e/deletecollection-intent-test" dcIntentActorAuthor = "Alice Liddell " + // dcIntentEditorAuthor is the identity that edits objects WITHOUT deleting them, so a removal + // credited to the last writer names it and is caught. + dcIntentEditorAuthor = "Cleanup Bot " ) // impersonatedConfigMapClient builds a clientset that impersonates asUser and carries @@ -296,6 +333,18 @@ func deleteConfigMapCollection(client *kubernetes.Clientset, ns, labelSelector s ) } +// annotateConfigMap writes a trivial change as the given identity, so that identity becomes the +// object's LAST WRITER without being the one who later deletes it. +func annotateConfigMap(client *kubernetes.Clientset, ns, name, key, value string) error { + patch := fmt.Sprintf(`{"metadata":{"annotations":{%q:%q}}}`, key, value) + _, err := client.CoreV1().ConfigMaps(ns).Patch( + context.Background(), name, k8stypes.MergePatchType, []byte(patch), metav1.PatchOptions{}) + if err != nil { + return fmt.Errorf("annotate configmap %s: %w", name, err) + } + return nil +} + func removeConfigMapFinalizers(client *kubernetes.Clientset, ns, name string) error { _, err := client.CoreV1().ConfigMaps(ns).Patch( context.Background(), @@ -337,6 +386,20 @@ func waitForFilePresent(repo *RepoArtifacts, repoPath string) { }, 2*time.Minute, 3*time.Second).Should(Succeed()) } +// waitForFileAuthoredBy asserts the file is present and its last commit carries the given author. +// It is how a spec proves who the CURRENT last writer is before changing that. +func waitForFileAuthoredBy(repo *RepoArtifacts, repoPath, author string) { + GinkgoHelper() + Eventually(func(g Gomega) { + pullLatestRepoState(g, repo.CheckoutDir) + _, statErr := os.Stat(filepath.Join(repo.CheckoutDir, repoPath)) + g.Expect(statErr).NotTo(HaveOccurred(), "file %s should be present in Git", repoPath) + got, logErr := gitRun(repo.CheckoutDir, "log", "-1", "--pretty=%an <%ae>", "--", repoPath) + g.Expect(logErr).NotTo(HaveOccurred()) + g.Expect(strings.TrimSpace(got)).To(Equal(author)) + }, 2*time.Minute, 3*time.Second).Should(Succeed()) +} + // waitForFileDeletedByActor asserts the file is gone from Git and the last commit that // touched its path is authored by the impersonated actor (dcIntentActorAuthor). func waitForFileDeletedByActor(repo *RepoArtifacts, repoPath string) { diff --git a/test/e2e/e2e_suite_test.go b/test/e2e/e2e_suite_test.go index 00a219bf..0d832b30 100644 --- a/test/e2e/e2e_suite_test.go +++ b/test/e2e/e2e_suite_test.go @@ -107,7 +107,10 @@ func assertNoAnomalousAuditOutcomes() { // older_than_high_water reorder, which is dropped/recovered and does NOT fail the gate) is // self-explaining in the artifacts. diag_all (when --audit-bytype-diag is on, as in e2e) holds // the full per-event records in Redis for deeper inspection. - for _, oc := range []string{"queued", "older_than_high_water", "non_numeric_rv", "rvless_empty_highwater", "not_needed", "shallow_dropped", "write_error"} { + for _, oc := range []string{ + "queued", "older_than_high_water", "non_numeric_rv", "rvless_empty_highwater", + "not_needed", "shallow_dropped", "no_attribution_fact", "write_error", + } { n, qErr := queryPrometheus(fmt.Sprintf( `sum(max_over_time(gitopsreverser_audit_events_total{outcome=%q}[2h])) or vector(0)`, oc)) if qErr == nil { @@ -168,35 +171,53 @@ func reportAttributionStats() { _, _ = fmt.Fprintf(GinkgoWriter, "\n📊 author attribution — %.0f resolutions this run\n", total) var absent float64 - for _, result := range []string{ - "exact_user", "exact_serviceaccount", "weak", "exact_deletecollection_item", "absent", + // The tiers, strongest first. collection_uid and collection_scope replaced the expander's + // exact_deletecollection_item: the collection match is two-tiered now, and the second tier + // resolves what used to degrade to committer-authored, so they are worth reading apart. + for _, tier := range []string{ + "exact", "collection_uid", "latest", "name", "collection_scope", "resource_version", "absent", } { n, qErr := queryPrometheus(fmt.Sprintf( - `sum(max_over_time(gitopsreverser_attribution_resolutions_total{result=%q}[2h])) or vector(0)`, result)) + `sum(max_over_time(gitopsreverser_attribution_resolutions_total{tier=%q}[2h])) or vector(0)`, tier)) if qErr != nil { continue } - if result == "absent" { + if tier == "absent" { absent = n } - _, _ = fmt.Fprintf(GinkgoWriter, " result %-28s = %6.0f (%5.1f%%)\n", result, n, 100*n/total) + _, _ = fmt.Fprintf(GinkgoWriter, " tier %-28s = %6.0f (%5.1f%%)\n", tier, n, 100*n/total) + } + + // Who the evidence named, which is now a label of its own rather than a value on two tiers. + for _, actorKind := range []string{"user", "serviceaccount", "none"} { + n, qErr := queryPrometheus(fmt.Sprintf( + `sum(max_over_time(gitopsreverser_attribution_resolutions_total{actor_kind=%q}[2h])) or vector(0)`, + actorKind)) + if qErr != nil { + continue + } + _, _ = fmt.Fprintf(GinkgoWriter, " actor_kind %-22s = %6.0f (%5.1f%%)\n", actorKind, n, 100*n/total) } // Cumulative histogram: le=X is "waited at most X seconds". // - // Split by result, because the two populations answer different questions. For RESOLVED - // results the tail says how close fact delivery ran to the window — anything above 3s only + // Split by tier, because the two populations answer different questions. For RESOLVED + // tiers the tail says how close fact delivery ran to the window — anything above 3s only // succeeded because e2e widens the grace past the 3s default. For "absent" the wait is just // the grace window being spent, so a cluster of absents at the ceiling means "waited the // whole time and nothing ever came", NOT "arrived slightly too late". + // + // The removal split is the one --author-attribution-grace is tuned from: a removal holds a + // fallback and keeps waiting for evidence about the deletion where a write does not. _, _ = fmt.Fprintf(GinkgoWriter, " wait distribution (cumulative):\n") - printWaitBuckets("resolved", `,result!="absent"`) - printWaitBuckets("absent ", `,result="absent"`) + printWaitBuckets("resolved", `,tier!="absent"`) + printWaitBuckets("absent ", `,tier="absent"`) + printWaitBuckets("removals", `,event_kind="removal"`) resolvedOverDefault, err := queryPrometheus( - `(sum(max_over_time(gitopsreverser_attribution_resolution_wait_seconds_count{result!="absent"}[2h])) ` + + `(sum(max_over_time(gitopsreverser_attribution_resolution_wait_seconds_count{tier!="absent"}[2h])) ` + `or vector(0)) - (sum(max_over_time(` + - `gitopsreverser_attribution_resolution_wait_seconds_bucket{result!="absent",le="3.0"}[2h])) or vector(0))`) + `gitopsreverser_attribution_resolution_wait_seconds_bucket{tier!="absent",le="3.0"}[2h])) or vector(0))`) if err == nil { _, _ = fmt.Fprintf(GinkgoWriter, " %.0f resolution(s) SUCCEEDED after waiting longer than the 3s default grace"+ @@ -217,7 +238,7 @@ func reportAttributionStats() { // OTel→Prometheus exporter as "1.0"/"3.0"/"10.0" — NOT "1"/"3"/"10" — so an integral boundary // must be formatted with a decimal or every query above 0.5 silently matches nothing and // returns 0, which reads as a plausible (and wrong) distribution rather than as an error. -func printWaitBuckets(label, resultSelector string) { +func printWaitBuckets(label, selector string) { var prev float64 for _, le := range attributionWaitBuckets { leLabel := strconv.FormatFloat(le, 'g', -1, 64) @@ -226,7 +247,7 @@ func printWaitBuckets(label, resultSelector string) { } n, qErr := queryPrometheus(fmt.Sprintf( `sum(max_over_time(gitopsreverser_attribution_resolution_wait_seconds_bucket{le=%q%s}[2h]))`+ - ` or vector(0)`, leLabel, resultSelector)) + ` or vector(0)`, leLabel, selector)) if qErr != nil { continue } @@ -255,14 +276,21 @@ func configuredAuthorModeEnabled() bool { return configuredAuthorModeFromArgs(out) } -// configuredAuthorModeFromArgs is the pure decision, mirroring the mode switch in -// cmd/main.go: attribution runs only when author attribution is on AND Redis is configured; -// anything else is configured-author mode. Both flags default to ENABLED in cmd/main.go -// (`--author-attribution` defaults true, `--redis-addr` defaults to "valkey:6379"), so an -// absent flag means attribution, and only an explicit opt-out turns it off. +// configuredAuthorModeFromArgs is the pure decision, mirroring the mode switch in cmd/main.go: +// attribution runs when author attribution is on AND it has a fact transport. Anything else is +// configured-author mode. Every flag defaults to ENABLED in cmd/main.go (`--author-attribution` +// defaults true, `--redis-addr` to "valkey:6379", the transport to redis), so an absent flag means +// attribution and only an explicit opt-out turns it off. +// +// "Has a transport" is the part that stopped meaning "has Redis". The in-memory transport needs no +// Redis at all, so an empty --redis-addr with --author-attribution-transport=memory is attribution +// running normally. Reading it as configured-author SKIPPED every attribution spec and reported a +// green run that had asserted nothing about attribution — the failure mode this probe exists to +// prevent, turned on the probe itself. func configuredAuthorModeFromArgs(args string) bool { attribution := true redisAddr := "valkey:6379" + transport := "redis" for _, arg := range strings.Fields(strings.NewReplacer(`"`, " ", `[`, " ", `]`, " ", `,`, " ").Replace(args)) { switch { case arg == "--author-attribution": @@ -279,9 +307,11 @@ func configuredAuthorModeFromArgs(args string) bool { } case strings.HasPrefix(arg, "--redis-addr="): redisAddr = strings.TrimPrefix(arg, "--redis-addr=") + case strings.HasPrefix(arg, "--author-attribution-transport="): + transport = strings.TrimPrefix(arg, "--author-attribution-transport=") } } - return !attribution || redisAddr == "" + return !attribution || (redisAddr == "" && transport != "memory") } var _ = AfterEach(func() { diff --git a/test/mutationlab/README.md b/test/mutationlab/README.md index 2e0ee7ac..dc96da47 100644 --- a/test/mutationlab/README.md +++ b/test/mutationlab/README.md @@ -31,7 +31,7 @@ gap (see [Capturing Intent, Not State](../../docs/spec/mutation-capture-lab-desi | 4 | No-op apply | `configmap_scenarios_test.go` · `TestNoOpApply` | `configmap/no-op-apply/` | audit, admission — **no** watch (resourceVersion unchanged) | | 5 | Status subresource | `workload_scenarios_test.go` · `TestStatusSubresource` | `deployment/status-update/` | watch ×2 — **no** audit, **no** admission | | 6 | Scale subresource | `workload_scenarios_test.go` · `TestScaleSubresource` | `deployment/scale-patch/` | watch, audit — **no** admission | -| 7 | Graceful delete | `workload_scenarios_test.go` · `TestGracefulDelete` | `pod/graceful-delete/` | watch (MODIFIED + DELETED), admission — **no** audit | +| 7 | Graceful delete (audit-EXCLUDED type) | `workload_scenarios_test.go` · `TestGracefulDelete` | `pod/graceful-delete/` | watch (MODIFIED + DELETED), admission — **no** audit ⚠️ | | 8 | Finalizer delete | `configmap_scenarios_test.go` · `TestFinalizerDelete` | `configmap/finalizer-delete/` | watch (MODIFIED + DELETED), audit (delete + patch — **no** second delete), admission (DELETE + UPDATE) | | 9 | Deletecollection | `configmap_scenarios_test.go` · `TestDeletecollection` | `configmap/deletecollection/` | watch ×N, audit ×1 (name-less), admission ×N (per object) | | 10 | Owner-ref cascade | `configmap_scenarios_test.go` · `TestOwnerRefCascade` | `configmap/owner-ref-cascade/` | watch DELETED ×2 (parent + cascaded child), audit ×2 (parent = human, child = `generic-garbage-collector`) | @@ -40,15 +40,32 @@ gap (see [Capturing Intent, Not State](../../docs/spec/mutation-capture-lab-desi | 13 | Optimistic-concurrency conflict | `configmap_scenarios_test.go` · `TestOptimisticConcurrencyConflict` | `configmap/conflict-update/` | audit ×1 (`update`, code 409) — **no** watch / **no** admission (rejected at storage, before admission) | | 14 | Multi-version CRD conversion | `crd_conversion_test.go` · `TestCRDConversion` | `widget/crd-conversion/` | watch (v2), audit (v1), admission (v1), conversion ×2 (both directions) | | 15 | Aggregated API write | `aggregated_api_test.go` · `TestAggregatedAPIWrite` | `flunder/aggregated-api-write/` | watch (full object), audit (empty body); admission is observed but not committed | +| 15a | Aggregated API delete | `aggregated_api_test.go` · `TestAggregatedAPIDelete` | `flunder/aggregated-api-delete/` | watch DELETED (full object), audit (`delete`, `objectRef` has the NAME from the URL but **no uid**) | +| 15b | Aggregated API deletecollection | `aggregated_api_test.go` · `TestAggregatedAPIDeletecollection` | `flunder/aggregated-api-deletecollection/` | watch ×N, audit ×1 (name-less, selector in `requestURI`, **no response body**), admission ×N (per object) | | 16 | Watch resync (`410 Gone`) | `watch_transport_test.go` · `TestWatchExpiredResourceVersion` | `configmap/watch-resync/` | watch ERROR (`Status` 410); driver verifies relist recovery | +| 18 | `generateName` create | `configmap_scenarios_test.go` · `TestGenerateNameCreate` | `configmap/generate-name-create/` | watch, audit (`objectRef` has **no name**, response body **does**), admission — the control for rows 15a/15b | | 17 | Bookmark | `watch_transport_test.go` · `TestWatchBookmark` | `configmap/watch-bookmark/` | watch BOOKMARK with resourceVersion | -All seventeen catalogued scenarios are now captured. Rows 16 and 17 test the watch -transport itself; the driver uses the lab's targeted `/watch-probe` endpoint so +All seventeen catalogued scenarios are now captured. Rows 15a and 15b are not catalog +rows: they extend row 15 to the removal verbs, because the create alone could not say +whether a proxied delete carries a uid (it does not) or whether a proxied +`deletecollection` returns a response body (it does not). Row 18 is their control: a +`generateName` create has an equally name-less `objectRef` and joins perfectly well, +because it carries a response body to recover the name from, which is what makes the +BODY rather than the name the thing the aggregated rows are actually missing. Rows 16 +and 17 test the watch transport itself; the driver uses the lab's targeted `/watch-probe` endpoint so transport-only events can be scenario-attributed — see the [watch-first ingestion architecture](../../docs/finished/watch-first-ingestion-architecture.md) design notes. +> ⚠️ **The "no audit" rows record this cluster's audit POLICY, not the API server.** The lab runs +> against the already-prepared e2e cluster and reuses its policy +> ([`test/e2e/cluster/audit/policy.yaml`](../e2e/cluster/audit/policy.yaml)), which drops `pods` and +> every `*/status` at `level: None` as runtime noise. So rows 5 and 7 show what a cluster configured +> like this one does not tell us — a `DELETE` on a pod is an audited request like any other when the +> policy asks for it. Read them as "excluded here", not as "unknowable". Re-capturing either row +> against a policy that includes those types would settle it by measurement. + ## How it integrates: swap the image, reuse the wiring The lab serves the **same** webhook URLs as the product — @@ -57,6 +74,14 @@ making a cluster capture with the lab is just swapping the controller image: no new audit policy, webhook config, or certificates. `task lab-e2e` does this on the already-prepared e2e cluster, then drives the scenarios serially. +Reusing the wiring means reusing whatever the cluster's audit kubeconfig points at, and +this one names its route: it posts to `/audit-webhook/default`, the "default" +ClusterProvider, because the bare path is the product's shared annotation-routed +endpoint. The lab therefore serves the whole `/audit-webhook/` subtree and records what +arrives whichever source it was addressed to. Serving only the exact bare path 404s every +event the cluster sends — which reads as a lab that captures watches and admission but +never a single audit record, in every scenario at once. + Row 15 (aggregated API write) is what settled the body-enrichment question: the official audit event for an aggregated-API write carries an empty body, yet the live watch carries the full object. Because the watch supplies the object content diff --git a/test/mutationlab/corpus/configmap/generate-name-create/admission.create.yaml b/test/mutationlab/corpus/configmap/generate-name-create/admission.create.yaml new file mode 100644 index 00000000..029b8933 --- /dev/null +++ b/test/mutationlab/corpus/configmap/generate-name-create/admission.create.yaml @@ -0,0 +1,62 @@ +request: + dryRun: false + kind: + group: "" + kind: ConfigMap + version: v1 + name: cm-gen- + namespace: + object: + apiVersion: v1 + data: + key: value + kind: ConfigMap + metadata: + creationTimestamp: + generateName: cm-gen- + labels: + mutationlab.configbutler.ai/scenario: generate-name-create + managedFields: + - apiVersion: v1 + fieldsType: FieldsV1 + fieldsV1: + f:data: + .: {} + f:key: {} + f:metadata: + f:generateName: {} + f:labels: + .: {} + f:mutationlab.configbutler.ai/scenario: {} + manager: e2e.test + operation: Update + time: + name: cm-gen- + namespace: + uid: + oldObject: null + operation: CREATE + options: + apiVersion: meta.k8s.io/v1 + kind: CreateOptions + requestKind: + group: "" + kind: ConfigMap + version: v1 + requestResource: + group: "" + resource: configmaps + version: v1 + resource: + group: "" + resource: configmaps + version: v1 + uid: + userInfo: + extra: + authentication.kubernetes.io/credential-id: + - + groups: + - system:masters + - system:authenticated + username: system:admin diff --git a/test/mutationlab/corpus/configmap/generate-name-create/audit.create.yaml b/test/mutationlab/corpus/configmap/generate-name-create/audit.create.yaml new file mode 100644 index 00000000..548aadce --- /dev/null +++ b/test/mutationlab/corpus/configmap/generate-name-create/audit.create.yaml @@ -0,0 +1,53 @@ +annotations: + authorization.k8s.io/decision: allow + authorization.k8s.io/reason: "" + validate-all.configbutler.ai/scenario: generate-name-create +auditID: +level: RequestResponse +objectRef: + apiVersion: v1 + namespace: + resource: configmaps +requestObject: + apiVersion: v1 + data: + key: value + kind: ConfigMap + metadata: + generateName: cm-gen- + labels: + mutationlab.configbutler.ai/scenario: generate-name-create + namespace: +requestReceivedTimestamp: +requestURI: /api/v1/namespaces//configmaps +responseObject: + apiVersion: v1 + data: + key: value + kind: ConfigMap + metadata: + creationTimestamp: + generateName: cm-gen- + labels: + mutationlab.configbutler.ai/scenario: generate-name-create + name: cm-gen- + namespace: + resourceVersion: + uid: +responseStatus: + code: 201 + metadata: {} +sourceIPs: +- +stage: ResponseComplete +stageTimestamp: +user: + extra: + authentication.kubernetes.io/credential-id: + - + groups: + - system:masters + - system:authenticated + username: system:admin +userAgent: e2e.test/v0.0.0 (linux/amd64) kubernetes/$Format +verb: create diff --git a/test/mutationlab/corpus/configmap/generate-name-create/watch.added.yaml b/test/mutationlab/corpus/configmap/generate-name-create/watch.added.yaml new file mode 100644 index 00000000..9b8f719e --- /dev/null +++ b/test/mutationlab/corpus/configmap/generate-name-create/watch.added.yaml @@ -0,0 +1,30 @@ +object: + apiVersion: v1 + data: + key: value + kind: ConfigMap + metadata: + creationTimestamp: + generateName: cm-gen- + labels: + mutationlab.configbutler.ai/scenario: generate-name-create + managedFields: + - apiVersion: v1 + fieldsType: FieldsV1 + fieldsV1: + f:data: + .: {} + f:key: {} + f:metadata: + f:generateName: {} + f:labels: + .: {} + f:mutationlab.configbutler.ai/scenario: {} + manager: e2e.test + operation: Update + time: + name: cm-gen- + namespace: + resourceVersion: + uid: +type: ADDED diff --git a/test/mutationlab/corpus/flunder/aggregated-api-delete/audit.delete.yaml b/test/mutationlab/corpus/flunder/aggregated-api-delete/audit.delete.yaml new file mode 100644 index 00000000..3f9bd75c --- /dev/null +++ b/test/mutationlab/corpus/flunder/aggregated-api-delete/audit.delete.yaml @@ -0,0 +1,30 @@ +annotations: + authorization.k8s.io/decision: allow + authorization.k8s.io/reason: "" +auditID: +level: RequestResponse +objectRef: + apiGroup: wardle.example.com + apiVersion: v1alpha1 + name: fl-del + namespace: + resource: flunders +requestReceivedTimestamp: +requestURI: /apis/wardle.example.com/v1alpha1/namespaces//flunders/fl-del +responseStatus: + code: 200 + metadata: {} +sourceIPs: +- +stage: ResponseComplete +stageTimestamp: +user: + extra: + authentication.kubernetes.io/credential-id: + - + groups: + - system:masters + - system:authenticated + username: system:admin +userAgent: e2e.test/v0.0.0 (linux/amd64) kubernetes/$Format +verb: delete diff --git a/test/mutationlab/corpus/flunder/aggregated-api-delete/watch.deleted.yaml b/test/mutationlab/corpus/flunder/aggregated-api-delete/watch.deleted.yaml new file mode 100644 index 00000000..d159b246 --- /dev/null +++ b/test/mutationlab/corpus/flunder/aggregated-api-delete/watch.deleted.yaml @@ -0,0 +1,30 @@ +object: + apiVersion: wardle.example.com/v1alpha1 + kind: Flunder + metadata: + creationTimestamp: + labels: + mutationlab.configbutler.ai/scenario: aggregated-api-delete + managedFields: + - apiVersion: wardle.example.com/v1alpha1 + fieldsType: FieldsV1 + fieldsV1: + f:metadata: + f:labels: + .: {} + f:mutationlab.configbutler.ai/scenario: {} + f:spec: + f:reference: {} + f:referenceType: {} + manager: e2e.test + operation: Update + time: + name: fl-del + namespace: + resourceVersion: + uid: + spec: + reference: doomed + referenceType: Flunder + status: {} +type: DELETED diff --git a/test/mutationlab/corpus/flunder/aggregated-api-deletecollection/admission.delete.fl-dc-a.yaml b/test/mutationlab/corpus/flunder/aggregated-api-deletecollection/admission.delete.fl-dc-a.yaml new file mode 100644 index 00000000..6eb5e127 --- /dev/null +++ b/test/mutationlab/corpus/flunder/aggregated-api-deletecollection/admission.delete.fl-dc-a.yaml @@ -0,0 +1,62 @@ +request: + dryRun: false + kind: + group: wardle.example.com + kind: Flunder + version: v1alpha1 + namespace: + object: null + oldObject: + apiVersion: wardle.example.com/v1alpha1 + kind: Flunder + metadata: + creationTimestamp: + labels: + mutationlab.configbutler.ai/scenario: aggregated-api-deletecollection + managedFields: + - apiVersion: wardle.example.com/v1alpha1 + fieldsType: FieldsV1 + fieldsV1: + f:metadata: + f:labels: + .: {} + f:mutationlab.configbutler.ai/scenario: {} + f:spec: + f:reference: {} + f:referenceType: {} + manager: e2e.test + operation: Update + time: + name: fl-dc-a + namespace: + resourceVersion: + uid: + spec: + reference: doomed + referenceType: Flunder + status: {} + operation: DELETE + options: + apiVersion: meta.k8s.io/v1 + kind: DeleteOptions + requestKind: + group: wardle.example.com + kind: Flunder + version: v1alpha1 + requestResource: + group: wardle.example.com + resource: flunders + version: v1alpha1 + resource: + group: wardle.example.com + resource: flunders + version: v1alpha1 + uid: + userInfo: + extra: + authentication.kubernetes.io/credential-id: + - + groups: + - system:masters + - system:authenticated + username: system:admin diff --git a/test/mutationlab/corpus/flunder/aggregated-api-deletecollection/admission.delete.fl-dc-b.yaml b/test/mutationlab/corpus/flunder/aggregated-api-deletecollection/admission.delete.fl-dc-b.yaml new file mode 100644 index 00000000..a4ef64ec --- /dev/null +++ b/test/mutationlab/corpus/flunder/aggregated-api-deletecollection/admission.delete.fl-dc-b.yaml @@ -0,0 +1,62 @@ +request: + dryRun: false + kind: + group: wardle.example.com + kind: Flunder + version: v1alpha1 + namespace: + object: null + oldObject: + apiVersion: wardle.example.com/v1alpha1 + kind: Flunder + metadata: + creationTimestamp: + labels: + mutationlab.configbutler.ai/scenario: aggregated-api-deletecollection + managedFields: + - apiVersion: wardle.example.com/v1alpha1 + fieldsType: FieldsV1 + fieldsV1: + f:metadata: + f:labels: + .: {} + f:mutationlab.configbutler.ai/scenario: {} + f:spec: + f:reference: {} + f:referenceType: {} + manager: e2e.test + operation: Update + time: + name: fl-dc-b + namespace: + resourceVersion: + uid: + spec: + reference: doomed + referenceType: Flunder + status: {} + operation: DELETE + options: + apiVersion: meta.k8s.io/v1 + kind: DeleteOptions + requestKind: + group: wardle.example.com + kind: Flunder + version: v1alpha1 + requestResource: + group: wardle.example.com + resource: flunders + version: v1alpha1 + resource: + group: wardle.example.com + resource: flunders + version: v1alpha1 + uid: + userInfo: + extra: + authentication.kubernetes.io/credential-id: + - + groups: + - system:masters + - system:authenticated + username: system:admin diff --git a/test/mutationlab/corpus/flunder/aggregated-api-deletecollection/admission.delete.fl-dc-c.yaml b/test/mutationlab/corpus/flunder/aggregated-api-deletecollection/admission.delete.fl-dc-c.yaml new file mode 100644 index 00000000..803c89de --- /dev/null +++ b/test/mutationlab/corpus/flunder/aggregated-api-deletecollection/admission.delete.fl-dc-c.yaml @@ -0,0 +1,62 @@ +request: + dryRun: false + kind: + group: wardle.example.com + kind: Flunder + version: v1alpha1 + namespace: + object: null + oldObject: + apiVersion: wardle.example.com/v1alpha1 + kind: Flunder + metadata: + creationTimestamp: + labels: + mutationlab.configbutler.ai/scenario: aggregated-api-deletecollection + managedFields: + - apiVersion: wardle.example.com/v1alpha1 + fieldsType: FieldsV1 + fieldsV1: + f:metadata: + f:labels: + .: {} + f:mutationlab.configbutler.ai/scenario: {} + f:spec: + f:reference: {} + f:referenceType: {} + manager: e2e.test + operation: Update + time: + name: fl-dc-c + namespace: + resourceVersion: + uid: + spec: + reference: doomed + referenceType: Flunder + status: {} + operation: DELETE + options: + apiVersion: meta.k8s.io/v1 + kind: DeleteOptions + requestKind: + group: wardle.example.com + kind: Flunder + version: v1alpha1 + requestResource: + group: wardle.example.com + resource: flunders + version: v1alpha1 + resource: + group: wardle.example.com + resource: flunders + version: v1alpha1 + uid: + userInfo: + extra: + authentication.kubernetes.io/credential-id: + - + groups: + - system:masters + - system:authenticated + username: system:admin diff --git a/test/mutationlab/corpus/flunder/aggregated-api-deletecollection/audit.deletecollection.yaml b/test/mutationlab/corpus/flunder/aggregated-api-deletecollection/audit.deletecollection.yaml new file mode 100644 index 00000000..3605e9f7 --- /dev/null +++ b/test/mutationlab/corpus/flunder/aggregated-api-deletecollection/audit.deletecollection.yaml @@ -0,0 +1,29 @@ +annotations: + authorization.k8s.io/decision: allow + authorization.k8s.io/reason: "" +auditID: +level: RequestResponse +objectRef: + apiGroup: wardle.example.com + apiVersion: v1alpha1 + namespace: + resource: flunders +requestReceivedTimestamp: +requestURI: /apis/wardle.example.com/v1alpha1/namespaces//flunders?labelSelector=mutationlab.configbutler.ai%2Fscenario%3Daggregated-api-deletecollection +responseStatus: + code: 200 + metadata: {} +sourceIPs: +- +stage: ResponseComplete +stageTimestamp: +user: + extra: + authentication.kubernetes.io/credential-id: + - + groups: + - system:masters + - system:authenticated + username: system:admin +userAgent: e2e.test/v0.0.0 (linux/amd64) kubernetes/$Format +verb: deletecollection diff --git a/test/mutationlab/corpus/flunder/aggregated-api-deletecollection/watch.deleted.fl-dc-a.yaml b/test/mutationlab/corpus/flunder/aggregated-api-deletecollection/watch.deleted.fl-dc-a.yaml new file mode 100644 index 00000000..11eda3b0 --- /dev/null +++ b/test/mutationlab/corpus/flunder/aggregated-api-deletecollection/watch.deleted.fl-dc-a.yaml @@ -0,0 +1,30 @@ +object: + apiVersion: wardle.example.com/v1alpha1 + kind: Flunder + metadata: + creationTimestamp: + labels: + mutationlab.configbutler.ai/scenario: aggregated-api-deletecollection + managedFields: + - apiVersion: wardle.example.com/v1alpha1 + fieldsType: FieldsV1 + fieldsV1: + f:metadata: + f:labels: + .: {} + f:mutationlab.configbutler.ai/scenario: {} + f:spec: + f:reference: {} + f:referenceType: {} + manager: e2e.test + operation: Update + time: + name: fl-dc-a + namespace: + resourceVersion: + uid: + spec: + reference: doomed + referenceType: Flunder + status: {} +type: DELETED diff --git a/test/mutationlab/corpus/flunder/aggregated-api-deletecollection/watch.deleted.fl-dc-b.yaml b/test/mutationlab/corpus/flunder/aggregated-api-deletecollection/watch.deleted.fl-dc-b.yaml new file mode 100644 index 00000000..5cee19d8 --- /dev/null +++ b/test/mutationlab/corpus/flunder/aggregated-api-deletecollection/watch.deleted.fl-dc-b.yaml @@ -0,0 +1,30 @@ +object: + apiVersion: wardle.example.com/v1alpha1 + kind: Flunder + metadata: + creationTimestamp: + labels: + mutationlab.configbutler.ai/scenario: aggregated-api-deletecollection + managedFields: + - apiVersion: wardle.example.com/v1alpha1 + fieldsType: FieldsV1 + fieldsV1: + f:metadata: + f:labels: + .: {} + f:mutationlab.configbutler.ai/scenario: {} + f:spec: + f:reference: {} + f:referenceType: {} + manager: e2e.test + operation: Update + time: + name: fl-dc-b + namespace: + resourceVersion: + uid: + spec: + reference: doomed + referenceType: Flunder + status: {} +type: DELETED diff --git a/test/mutationlab/corpus/flunder/aggregated-api-deletecollection/watch.deleted.fl-dc-c.yaml b/test/mutationlab/corpus/flunder/aggregated-api-deletecollection/watch.deleted.fl-dc-c.yaml new file mode 100644 index 00000000..846e78bd --- /dev/null +++ b/test/mutationlab/corpus/flunder/aggregated-api-deletecollection/watch.deleted.fl-dc-c.yaml @@ -0,0 +1,30 @@ +object: + apiVersion: wardle.example.com/v1alpha1 + kind: Flunder + metadata: + creationTimestamp: + labels: + mutationlab.configbutler.ai/scenario: aggregated-api-deletecollection + managedFields: + - apiVersion: wardle.example.com/v1alpha1 + fieldsType: FieldsV1 + fieldsV1: + f:metadata: + f:labels: + .: {} + f:mutationlab.configbutler.ai/scenario: {} + f:spec: + f:reference: {} + f:referenceType: {} + manager: e2e.test + operation: Update + time: + name: fl-dc-c + namespace: + resourceVersion: + uid: + spec: + reference: doomed + referenceType: Flunder + status: {} +type: DELETED diff --git a/test/mutationlab/e2e/aggregated_api_test.go b/test/mutationlab/e2e/aggregated_api_test.go index 06f259ce..0bf4efe1 100644 --- a/test/mutationlab/e2e/aggregated_api_test.go +++ b/test/mutationlab/e2e/aggregated_api_test.go @@ -136,3 +136,206 @@ func flunderReference(r *mutationlab.Record) string { } return env.Object.Spec.Reference } + +// TestAggregatedAPIDelete is the delete half of Row 15, and it exists because the create half +// raised a question it could not answer: an aggregated-API write is audited with an EMPTY body, so +// what does an aggregated-API DELETE look like — is it audited at all, does its objectRef carry a +// uid, and does the watch DELETED carry the object? +// +// The answer decides whether a removal of an aggregated type can be attributed to its deleter at +// all. The join needs either a uid on the fact (which the exact and latest tiers key on) or a +// collection fact covering it. If a proxied delete is audited with no uid, per-object attribution +// for aggregated types is structurally impossible and only the collection tiers can ever work. +func TestAggregatedAPIDelete(t *testing.T) { + h := newHarness(t) + ctx := context.Background() + s := h.newScenario(ctx, t, "aggregated-api-delete") + + flunder := &unstructured.Unstructured{Object: map[string]any{ + "apiVersion": "wardle.example.com/v1alpha1", + "kind": "Flunder", + "metadata": map[string]any{ + "name": "fl-del", + "namespace": s.ns, + "labels": map[string]any{scenarioLabel: s.id}, + }, + "spec": map[string]any{"referenceType": "Flunder", "reference": "doomed"}, + }} + if _, err := h.dyn.Resource(flunderGVR).Namespace(s.ns).Create(ctx, flunder, metav1.CreateOptions{}); err != nil { + t.Fatalf("create flunder: %v", err) + } + // The create's own records are setup, not the finding. Union the namespace because an + // aggregated audit event carries no object label to attribute by. + h.drain(t, s.id, drainSpec{ + minCount: 1, settle: 3 * time.Second, timeout: 90 * time.Second, alsoNamespace: s.ns, + until: func(rs []mutationlab.Record) bool { + return flunderRecordNamed(rs, mutationlab.SourceWatch, "ADDED", "fl-del") != nil + }, + }) + h.clearRecords(t) + + if err := h.dyn.Resource(flunderGVR).Namespace(s.ns). + Delete(ctx, "fl-del", metav1.DeleteOptions{}); err != nil { + t.Fatalf("delete flunder: %v", err) + } + + records := h.drain(t, s.id, drainSpec{ + minCount: 1, settle: 5 * time.Second, timeout: 90 * time.Second, alsoNamespace: s.ns, + until: func(rs []mutationlab.Record) bool { + return flunderRecordNamed(rs, mutationlab.SourceWatch, "DELETED", "fl-del") != nil + }, + }) + + deleted := flunderRecordNamed(records, mutationlab.SourceWatch, "DELETED", "fl-del") + audit := flunderRecordNamed(records, mutationlab.SourceAudit, "", "fl-del") + if deleted == nil { + t.Fatal("no watch DELETED for the flunder; the aggregated-API watch did not carry the removal") + } + + // THE FINDING, whichever way it goes. A delete that is audited WITH a uid can be attributed + // per object; audited WITHOUT one can only ever be reached by the collection tiers; not + // audited at all can never be attributed, exactly like a type the policy excludes. + switch { + case audit == nil: + t.Logf("FINDING: an aggregated-API delete produced NO audit record at all. Per-object "+ + "attribution of %s removals is impossible; every one ships committer-authored.", + flunderGVR.Resource) + case audit.Key.UID == "": + t.Logf("FINDING: the aggregated-API delete IS audited (verb=%q) but its objectRef carries "+ + "no uid, so the exact and latest tiers can never match it. Only a collection fact can "+ + "attribute an aggregated removal.", audit.Summary.Operation) + default: + t.Logf("FINDING: the aggregated-API delete is audited with uid %q, so per-object "+ + "attribution works for aggregated types after all.", audit.Key.UID) + } + t.Logf("watch DELETED carries object=%v; audit present=%v", deleted.Summary.HasObject, audit != nil) + + commit := []mutationlab.Record{*deleted} + if audit != nil { + commit = append([]mutationlab.Record{*audit}, commit...) + } + h.syncCorpus(t, "flunder/aggregated-api-delete", commit) +} + +// TestAggregatedAPIDeletecollection is the case the whole collection-fact design turns on for +// aggregated types, and the one a hand-written fixture cannot settle. +// +// The kube-apiserver PROXIES the request to the extension server and never decodes the response it +// streamed back, so the expectation is an audited deletecollection with NO response body: no uid +// set, and therefore a join that can only proceed by SCOPE — type, namespace, selector, window. +// That is exactly the case the deleted response-body expander produced nothing for. +// +// If the body IS present, the join upgrades itself to uid membership and the scope tier is not +// needed here. Either result is worth having in the corpus; the point is to stop inferring it. +func TestAggregatedAPIDeletecollection(t *testing.T) { + h := newHarness(t) + ctx := context.Background() + s := h.newScenario(ctx, t, "aggregated-api-deletecollection") + + // Three, not two. The finding is an asymmetry — N object removals against ONE name-less audit + // event with no response body — and two of something reads as a pair, which is the one count + // that could still be mistaken for a coincidence. Three makes the fan-out unmistakable in the + // corpus: three watch DELETEDs, three admissions, one audit record naming none of them. + names := []string{"fl-dc-a", "fl-dc-b", "fl-dc-c"} + for _, name := range names { + flunder := &unstructured.Unstructured{Object: map[string]any{ + "apiVersion": "wardle.example.com/v1alpha1", + "kind": "Flunder", + "metadata": map[string]any{ + "name": name, + "namespace": s.ns, + "labels": map[string]any{scenarioLabel: s.id}, + }, + "spec": map[string]any{"referenceType": "Flunder", "reference": "doomed"}, + }} + if _, err := h.dyn.Resource(flunderGVR).Namespace(s.ns). + Create(ctx, flunder, metav1.CreateOptions{}); err != nil { + t.Fatalf("create %s: %v", name, err) + } + } + h.drain(t, s.id, drainSpec{ + minCount: len(names), settle: 3 * time.Second, timeout: 90 * time.Second, alsoNamespace: s.ns, + until: func(rs []mutationlab.Record) bool { + return flunderRecordNamed(rs, mutationlab.SourceWatch, "ADDED", names[len(names)-1]) != nil + }, + }) + h.clearRecords(t) + + selector := metav1.ListOptions{LabelSelector: scenarioLabel + "=" + s.id} + if err := h.dyn.Resource(flunderGVR).Namespace(s.ns). + DeleteCollection(ctx, metav1.DeleteOptions{}, selector); err != nil { + t.Fatalf("deletecollection flunders: %v", err) + } + + records := h.drain(t, s.id, drainSpec{ + minCount: len(names), settle: 5 * time.Second, timeout: 90 * time.Second, alsoNamespace: s.ns, + until: func(rs []mutationlab.Record) bool { + return flunderRecordNamed(rs, mutationlab.SourceWatch, "DELETED", names[len(names)-1]) != nil + }, + }) + + // The asymmetry the collection fact exists for: N watch removals against ONE audit event. + watches := 0 + for i := range records { + r := &records[i] + if r.Source == mutationlab.SourceWatch && r.Summary.WatchType == "DELETED" && isFlunder(r) { + watches++ + } + } + if watches != len(names) { + t.Errorf("aggregated deletecollection produced %d watch DELETEDs; want %d (per-object fan-out)", + watches, len(names)) + } + + audit := flunderRecordNamed(records, mutationlab.SourceAudit, "", "") + switch { + case audit == nil: + t.Log("FINDING: an aggregated deletecollection produced NO audit record. Nothing can " + + "attribute these removals; every one ships committer-authored.") + case !audit.Summary.HasResponseObject: + t.Log("FINDING: the aggregated deletecollection IS audited with NO response body, so the " + + "fact carries no uid set and the join must fall back to SCOPE matching. This is the " + + "case the deleted response-body expander produced nothing at all for.") + default: + t.Log("FINDING: the aggregated deletecollection returned a response body, so the fact " + + "carries the uid set and the join resolves by uid membership.") + } + t.Logf("%d watch DELETEDs against %d audit record(s)", watches, countSource(records, mutationlab.SourceAudit)) + + h.syncCorpus(t, "flunder/aggregated-api-deletecollection", records) +} + +// flunderRecordNamed is flunderRecord with an optional name filter, so a scenario that creates more +// than one flunder can pick out the one it means. An empty name matches any flunder record. +func flunderRecordNamed( + records []mutationlab.Record, + src mutationlab.Source, + watchType, name string, +) *mutationlab.Record { + for i := range records { + r := &records[i] + if r.Source != src { + continue + } + if watchType != "" && r.Summary.WatchType != watchType { + continue + } + if !isFlunder(r) { + continue + } + if name != "" && r.Key.Name != name { + continue + } + return r + } + return nil +} + +// isFlunder says whether a record is about a flunder, and it cannot key on the resource alone: +// only audit and admission records carry a plural `resource`, because only they are built from a +// request. A watch record's key comes from the object's GroupVersionKind, so its `resource` is +// always empty — filtering on it drops exactly the watch events these scenarios are measuring. +func isFlunder(r *mutationlab.Record) bool { + return r.Key.Resource == flunderGVR.Resource || + (r.Key.Group == flunderGVR.Group && r.Key.Version == flunderGVR.Version) +} diff --git a/test/mutationlab/e2e/configmap_scenarios_test.go b/test/mutationlab/e2e/configmap_scenarios_test.go index 329487a7..31d8634d 100644 --- a/test/mutationlab/e2e/configmap_scenarios_test.go +++ b/test/mutationlab/e2e/configmap_scenarios_test.go @@ -60,6 +60,79 @@ func TestCreateSucceeds(t *testing.T) { h.syncCorpus(t, "configmap/create-succeeds", records) } +// TestGenerateNameCreate captures Row 18, and it exists to settle by measurement a claim that was +// only ever reasoning: that an empty `objectRef.name` is survivable as long as the event carries a +// BODY to recover the name from. +// +// Two very different populations produce an audit event whose objectRef has no name. A create with +// metadata.generateName has none because the API server assigns the name after the request is +// written; an aggregated-API write has none because the API server proxied the request and never saw +// the object. It would be reasonable to expect both to fail the same way. They do not, and the +// discriminator is the body rather than the name: `IdentityFromAuditEvent` backfills the missing name +// and uid from the response object, which a generateName create has and an aggregated write does not +// (corpus flunder/aggregated-api-write, where the body is empty). +// +// So this row is the CONTROL for the aggregated rows. It asserts the recovery material is actually +// present rather than assuming it, and the corpus then carries both halves side by side. +func TestGenerateNameCreate(t *testing.T) { + h := newHarness(t) + ctx := context.Background() + s := h.newScenario(ctx, t, "generate-name-create") + + meta := s.meta("") + meta.GenerateName = "cm-gen-" + created, err := h.kube.CoreV1().ConfigMaps(s.ns).Create(ctx, + &corev1.ConfigMap{ObjectMeta: meta, Data: map[string]string{"key": "value"}}, metav1.CreateOptions{}) + if err != nil { + t.Fatalf("create configmap with generateName: %v", err) + } + if created.Name == "" { + t.Fatal("the API server returned no assigned name") + } + + records := h.drain(t, s.id, drainSpec{minCount: 3, settle: 2 * time.Second, timeout: 60 * time.Second}) + + var audit *mutationlab.Record + for i := range records { + if records[i].Source == mutationlab.SourceAudit { + audit = &records[i] + break + } + } + if audit == nil { + t.Fatal("no audit event for the generateName create") + } + + // THE RESULT, in two halves. The objectRef cannot name the object, and the body can — which is + // exactly the asymmetry that makes this population joinable where the aggregated one is not. + if auditObjectRefName(audit) != "" { + t.Errorf("expected the audit objectRef to carry no name for a generateName create, got %q", + auditObjectRefName(audit)) + } + if !audit.Summary.HasResponseObject { + t.Error("the generateName create's audit event carried no response object, so the assigned " + + "name and uid could not be recovered; this is the aggregated failure mode on an ordinary type") + } + t.Logf("Row 18: assigned name=%q; audit objectRef name=%q; hasRequestObject=%v hasResponseObject=%v", + created.Name, auditObjectRefName(audit), audit.Summary.HasRequestObject, audit.Summary.HasResponseObject) + + h.syncCorpus(t, "configmap/generate-name-create", records) +} + +// auditObjectRefName reads objectRef.name off a raw audit event, the field that is empty for both a +// generateName create and an aggregated-API write. +func auditObjectRefName(r *mutationlab.Record) string { + var event struct { + ObjectRef struct { + Name string `json:"name"` + } `json:"objectRef"` + } + if err := json.Unmarshal(r.Raw, &event); err != nil { + return "" + } + return event.ObjectRef.Name +} + // TestUpdate captures Row 2: an Update (PUT) after a create. The create is set up // and cleared, so the corpus is just the update moment — admission UPDATE, audit // update, watch MODIFIED — with the verb that differs by request shape. diff --git a/test/mutationlab/e2e/workload_scenarios_test.go b/test/mutationlab/e2e/workload_scenarios_test.go index ea047111..980eeb3c 100644 --- a/test/mutationlab/e2e/workload_scenarios_test.go +++ b/test/mutationlab/e2e/workload_scenarios_test.go @@ -165,12 +165,24 @@ func gracefulPod(s scenario, name string) *corev1.Pod { // disappear (watch DELETED). A non-graceful delete would skip straight to // DELETED, so the lingering MODIFIED is the behavior under test. // -// Pods are dropped from the audit policy entirely, so there is no audit record; -// pods are top-level, so the DELETE does reach the validating webhook. The corpus -// keeps the two semantically load-bearing watch moments (the deletion-pending -// MODIFIED and the terminal DELETED) plus the admission DELETE; the intermediate -// kubelet status writes during termination are timing-dependent, so they are -// asserted as a law over the full drain rather than committed as flaky moments. +// Pods are dropped from THIS CLUSTER'S AUDIT POLICY entirely, so there is no audit record; pods are +// top-level, so the DELETE does reach the validating webhook. The corpus keeps the two semantically +// load-bearing watch moments (the deletion-pending MODIFIED and the terminal DELETED) plus the +// admission DELETE; the intermediate kubelet status writes during termination are timing-dependent, +// so they are asserted as a law over the full drain rather than committed as flaky moments. +// +// **Read the audit silence as a POLICY, not as a property of Kubernetes.** A DELETE on a pod is an +// audited request like any other; this row shows zero audit records because +// test/e2e/cluster/audit/policy.yaml lists pods at level: None as runtime noise. Design records +// have twice generalised this row into "a graceful pod delete produces no audit event at all" and +// built arguments on it, so the distinction is worth stating where the measurement is taken. What +// this row actually demonstrates is the shape of an AUDIT-EXCLUDED type, which is the population +// that costs the attribution resolver its whole grace window on every removal +// (docs/design/attribution-removal-wait-options.md). +// +// The audited equivalent of this two-step removal is TestFinalizerDelete: a configmap held by a +// finalizer takes the same deletionTimestamp-then-DELETED path and IS audited, so that is the row +// to read for what a graceful removal looks like when the policy asks for it. func TestGracefulDelete(t *testing.T) { h := newHarness(t) ctx := context.Background()