feature: Webhook: target Pods at CREATE time instead of workload objects

[cwiklik]

Feature Description

Summary

Change the mutating webhook from targeting workload objects (Deployments, StatefulSets, DaemonSets, Jobs, CronJobs) to targeting Pods at CREATE time. This is the pattern used by Istio, Linkerd, and Vault Agent Injector, and eliminates GitOps drift caused by mutating the workload object stored in etcd.

This change is part of the Summit MVP for the CR-triggered injection model (see design proposal PR #770).

Motivation

The current webhook intercepts workload objects and mutates the PodTemplateSpec inside them. This causes two problems:

1. GitOps drift: Argo CD, Flux, and other GitOps tools compare the workload object in Git with the object in the cluster. The webhook-injected sidecars appear in the cluster's Deployment but not in Git, triggering drift detection and potential revert loops.

2. Ownership conflict: The Deployment object is owned by the developer (or GitOps tool). Mutating it at admission time mixes platform concerns (sidecar injection) into a developer-owned resource.

Targeting Pods at CREATE time solves both:

• The Deployment in etcd stays exactly as the developer defined it
• Sidecars are only visible at the Pod level (kubectl get pod -o yaml)
• GitOps tools see no drift because the Deployment object is unchanged

Current behavior

MutatingWebhookConfiguration (config/webhook/manifests.yaml):

• rules.resources: deployments, statefulsets, daemonsets, jobs, cronjobs
• rules.apiGroups: apps, batch
• rules.operations: CREATE, UPDATE

Webhook handler (internal/webhook/v1alpha1/authbridge_webhook.go):

• Handle() switches on req.Kind.Kind (Deployment, StatefulSet, DaemonSet, Job, CronJob)
• Extracts PodSpec from each workload type's template
• Passes PodSpec to injector.PodMutator.InjectAuthBridge()
• Marshals the mutated workload object back

Proposed Solution

Required changes

1. MutatingWebhookConfiguration

File: config/webhook/manifests.yaml

• Change rules.resources from deployments;statefulsets;daemonsets;jobs;cronjobs to pods
• Change rules.apiGroups from apps;batch to "" (core)
• Change rules.operations from CREATE;UPDATE to CREATE only (Pods are immutable after creation — no need for UPDATE)
• Add objectSelector to match only pods with kagenti.io/type label:

  objectSelector:
    matchExpressions:
    - key: kagenti.io/type
      operator: In
      values: ["agent", "tool"]
    - key: kagenti.io/inject
      operator: NotIn
      values: ["disabled"]

• Keep existing namespaceSelector (exclude kube-system, etc.)
• Update the reinvocationPolicy to IfNeeded (other webhooks may modify the Pod before ours runs)

2. Webhook handler

File: internal/webhook/v1alpha1/authbridge_webhook.go

• Simplify Handle() — remove the switch on Kind, decode only corev1.Pod
• Extract PodSpec directly from the Pod object (no more navigating workload templates)
• Extract labels from pod.Labels (not from template labels)
• Pass PodSpec to injector.PodMutator.InjectAuthBridge() as before
• Marshal the mutated Pod back
• Update idempotency check (isAlreadyInjected) — works the same on Pod's containers
• Update kubebuilder marker comment to reflect new resource targeting
• Remove imports for appsv1 and batchv1 (no longer needed)

3. AgentRuntime CR lookup (new)

File: internal/webhook/v1alpha1/authbridge_webhook.go or new file

• At admission time, look up an AgentRuntime CR that targets the Pod's owner workload:
  • Read ownerReferences from the Pod to find the ReplicaSet
  • Read ownerReferences from the ReplicaSet to find the Deployment
  • Query for an AgentRuntime CR with targetRef matching the Deployment
• If AgentRuntime CR found: merge CR config with ConfigMap defaults, use merged config for injection
• If AgentRuntime CR not found: use ConfigMap defaults only
• Add RBAC: get/list AgentRuntime CRs, get/list ReplicaSets (for owner chain resolution)
• Cache AgentRuntime CRs using an informer to avoid per-admission API calls

4. Injector updates

File: internal/webhook/injector/pod_mutator.go

• Review InjectAuthBridge() — it already operates on *corev1.PodSpec, so minimal changes expected
• Update resourceName parameter: currently receives the workload name, now receives the Pod name. Consider whether sidecar configuration (e.g., client registration name template) needs the workload name instead — if so, pass it from the owner chain resolution
• Verify volume mounts and init container injection work correctly on a Pod object (vs. embedded PodSpec in a Deployment)

5. Tests

Files: internal/webhook/v1alpha1/webhook_suite_test.go, internal/webhook/injector/pod_mutator_test.go

• Update webhook handler tests: send Pod admission requests instead of Deployment requests
• Test: Pod with kagenti.io/type: agent label → sidecars injected
• Test: Pod with kagenti.io/type: tool label → sidecars injected (when feature gate enabled)
• Test: Pod without kagenti.io/type label → no injection (should not reach webhook due to objectSelector, but test defense-in-depth)
• Test: Pod with kagenti.io/inject: disabled → no injection
• Test: Pod with AgentRuntime CR targeting its owner → config merged from CR
• Test: Pod without AgentRuntime CR → config from defaults only
• Test: already-injected Pod → skipped (idempotency)
• Update or remove tests for Deployment/StatefulSet/DaemonSet/Job/CronJob admission (no longer applicable)
• make test passes

6. Helm chart

Files: charts/kagenti-webhook/

• Update MutatingWebhookConfiguration template to target Pods
• Update RBAC templates for new permissions (AgentRuntime CRs, ReplicaSets)
• Verify helm template renders correctly
• Verify helm install on Kind cluster works

7. Documentation

• Update ARCHITECTURE.md to reflect Pod-level targeting
• Update README.md if it references the webhook targeting model
• Add note about Argo CD compatibility (no longer causes drift on Deployments)

Progress tracker

Phase 1: Core webhook change

• MutatingWebhookConfiguration targets Pods (Feature: Create ToolHive Webhook Extension Project #1)
• Webhook handler simplified to decode Pods only (Feature: Consolidate Agent and MCPServer Webhooks to Avoid Code Duplication #2)
• Injector verified for Pod-level operation (feature: Remove initContainer and Replace it with Kagenti Sidecar Containers #4)
• Basic tests passing (Add Helm Chart and CI/CD Pipeline #5)

Phase 2: AgentRuntime CR integration

• Owner chain resolution (Pod → ReplicaSet → Deployment) (bug: Replace initContainer if Favor of client-registration and spiffy-helper Containers #3)
• AgentRuntime CR lookup and config merging (bug: Replace initContainer if Favor of client-registration and spiffy-helper Containers #3)
• AgentRuntime CR informer/cache (bug: Replace initContainer if Favor of client-registration and spiffy-helper Containers #3)
• Integration tests for CR lookup (Add Helm Chart and CI/CD Pipeline #5)

Phase 3: Packaging and docs

• Helm chart updated (Consolidate sidecar injection into shared webhook logic #6)
• Documentation updated (Bump sigs.k8s.io/controller-runtime from 0.22.1 to 0.22.3 in /kagenti-webhook #7)
• E2E verified on Kind cluster

Dependencies

Dependency	Repo	Notes
AgentRuntime CRD must exist	kagenti-operator	Webhook needs to query AgentRuntime CRs. CRD must be installed first.
AgentRuntime controller applies labels	kagenti-operator	Labels on PodTemplateSpec trigger Pod creation, which triggers the webhook.

Want to contribute?

• I would like to work on this issue.

Additional Context

No response