Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions kagenti-operator/cmd/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,7 @@ func main() {
var enableMLflow bool

var spireTrustDomain string
var spireIDTemplate string
var spireTrustBundleConfigMapName string
var spireTrustBundleConfigMapNS string
var spireTrustBundleConfigMapKey string
Expand Down Expand Up @@ -131,6 +132,14 @@ func main() {

flag.StringVar(&spireTrustDomain, "spire-trust-domain", "",
"SPIRE trust domain for identity binding (e.g. 'example.org')")

spireIDTemplateDefault := "spiffe://{{.TrustDomain}}/ns/{{.Namespace}}/sa/{{.ServiceAccount}}"
if envTemplate := os.Getenv("SPIRE_ID_TEMPLATE"); envTemplate != "" {
spireIDTemplateDefault = envTemplate
}
flag.StringVar(&spireIDTemplate, "spire-id-template", spireIDTemplateDefault,
"SPIFFE ID template for Keycloak client IDs (must match ClusterSPIFFEID spiffeIDTemplate, "+
"can also be set via SPIRE_ID_TEMPLATE env var)")
flag.StringVar(&spireTrustBundleConfigMapName, "spire-trust-bundle-configmap", "",
"Name of the ConfigMap containing the SPIRE trust bundle (SPIFFE JSON format)")
flag.StringVar(&spireTrustBundleConfigMapNS, "spire-trust-bundle-configmap-namespace", "",
Expand Down Expand Up @@ -404,6 +413,7 @@ func main() {
APIReader: mgr.GetAPIReader(),
Scheme: mgr.GetScheme(),
SpireTrustDomain: spireTrustDomain,
SpireIDTemplate: spireIDTemplate,
KeycloakAdminTokenCache: &keycloak.CachedAdminTokenProvider{},
}).SetupWithManager(mgr); err != nil {
setupLog.Error(err, "unable to create controller", "controller", "ClientRegistration")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ Other workloads are ignored by this controller.
3. Read **`keycloak-admin-secret`** (admin username/password).
4. Compute **Keycloak client ID**:
- If `SPIRE_ENABLED` is not true: `namespace/workloadName`.
- If SPIRE is enabled: `spiffe://<trust-domain>/ns/<namespace>/sa/<serviceAccount>` (requires a **non-default** `serviceAccountName` and operator **`--spire-trust-domain`**).
- If SPIRE is enabled: render the `--spire-id-template` with trust domain, namespace, and service account (default format: `spiffe://<trust-domain>/ns/<namespace>/sa/<serviceAccount>`). Requires a **non-default** `serviceAccountName`, operator **`--spire-trust-domain`**, and **`--spire-id-template`**.
5. **Register or fetch** the client via Keycloak admin API (`internal/keycloak`).
6. **Create or update** the credentials Secret; set **owner** to the Deployment/StatefulSet.
7. **Patch** the pod template annotation `kagenti.io/keycloak-client-credentials-secret-name` to the deterministic secret name.
Expand All @@ -88,6 +88,7 @@ Other workloads are ignored by this controller.
|-----------|-------------|------|
| Operator (controller) | `--enable-operator-client-registration` (default **false**) | Master switch for the ClientRegistration controller. |
| Operator (controller) | `--spire-trust-domain` | Required for SPIFFE-shaped client IDs when `authbridge-config` has `SPIRE_ENABLED=true`. |
| Operator (controller) | `--spire-id-template` (default `spiffe://{{.TrustDomain}}/ns/{{.Namespace}}/sa/{{.ServiceAccount}}`) | SPIFFE ID template for Keycloak client IDs. Must match the `spiffeIDTemplate` in ClusterSPIFFEID resources. Can also be set via `SPIRE_ID_TEMPLATE` environment variable. |
| Operator (webhook) | `--enable-client-registration` | Cluster-wide gate for client-registration **injection** (precedence still applies). |
| Operator (webhook) | Feature gates file (`/etc/kagenti/feature-gates/feature-gates.yaml`) | `clientRegistration`, `injectTools`, `globalEnabled`, etc., same as for injected sidecars. |

Expand All @@ -110,6 +111,7 @@ Other workloads are ignored by this controller.
### 3.3 Operator configuration

- When `authbridge-config` sets `SPIRE_ENABLED=true`, configure **`--spire-trust-domain`** to match the SPIRE server trust domain (same value as used for workload SPIFFE IDs).
- Configure **`--spire-id-template`** (or set the **`SPIRE_ID_TEMPLATE`** environment variable) to match the `spiffeIDTemplate` used in your ClusterSPIFFEID resources. The default template `spiffe://{{.TrustDomain}}/ns/{{.Namespace}}/sa/{{.ServiceAccount}}` matches the standard format. If you use a custom template in SPIRE (e.g., with pod name or labels), update this flag or environment variable accordingly. The flag takes precedence over the environment variable if both are set.
- Ensure the operator can read **`authbridge-config`** and **`keycloak-admin-secret`** in agent namespaces, and create/update **`kagenti-keycloak-client-credentials-*`** Secrets there (see RBAC below).

### 3.4 RBAC: why Secret rules are cluster-wide
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,13 @@ you may not use this file except in compliance with the License.
package controller

import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"fmt"
"strings"
"text/template"
"time"

appsv1 "k8s.io/api/apps/v1"
Expand Down Expand Up @@ -61,6 +63,7 @@ type ClientRegistrationReconciler struct {
Scheme *runtime.Scheme

SpireTrustDomain string
SpireIDTemplate string
// KeycloakAdminTokenCache caches admin password-grant tokens by Keycloak URL and credentials to
// avoid a token request on every reconcile. If nil, PasswordGrantToken is used without caching.
KeycloakAdminTokenCache *keycloak.CachedAdminTokenProvider
Expand Down Expand Up @@ -180,7 +183,7 @@ func (r *ClientRegistrationReconciler) reconcileOne(

spireEnabled := strings.EqualFold(strings.TrimSpace(ab.SpireEnabled), "true")
clientName := ns + "/" + workloadName
clientID, err := resolveKeycloakClientID(ns, workloadName, template.Spec.ServiceAccountName, spireEnabled, r.SpireTrustDomain)
clientID, err := resolveKeycloakClientID(ns, workloadName, template.Spec.ServiceAccountName, spireEnabled, r.SpireTrustDomain, r.SpireIDTemplate)
if err != nil {
logger.Info("cannot resolve Keycloak client id yet", "reason", err.Error())
return ctrl.Result{RequeueAfter: 30 * time.Second}, nil
Expand Down Expand Up @@ -375,7 +378,7 @@ func readClusterFeatureGates(ctx context.Context, c client.Reader) (globalOn, cl
return globalOn, clientReg, injectTools, nil
}

func resolveKeycloakClientID(namespace, workloadName, serviceAccount string, spireEnabled bool, trustDomain string) (string, error) {
func resolveKeycloakClientID(namespace, workloadName, serviceAccount string, spireEnabled bool, trustDomain, spireIDTemplate string) (string, error) {
sa := strings.TrimSpace(serviceAccount)
if sa == "" {
sa = "default"
Expand All @@ -389,7 +392,28 @@ func resolveKeycloakClientID(namespace, workloadName, serviceAccount string, spi
if trustDomain == "" {
return "", fmt.Errorf("SPIRE enabled: operator --spire-trust-domain is required for operator-managed client registration")
}
return fmt.Sprintf("spiffe://%s/ns/%s/sa/%s", trustDomain, namespace, sa), nil
if spireIDTemplate == "" {
return "", fmt.Errorf("SPIRE enabled: operator --spire-id-template is required for operator-managed client registration")
}

// Parse and render the SPIFFE ID template
tmpl, err := template.New("spiffe-id").Parse(spireIDTemplate)
if err != nil {
return "", fmt.Errorf("invalid SPIFFE ID template: %w", err)
}

data := map[string]string{
"TrustDomain": trustDomain,
"Namespace": namespace,
"ServiceAccount": sa,
}

var buf bytes.Buffer
if err := tmpl.Execute(&buf, data); err != nil {
return "", fmt.Errorf("failed to render SPIFFE ID template: %w", err)
}

return buf.String(), nil
}

func keycloakClientCredentialsSecretName(namespace, workload string) string {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -119,18 +119,27 @@ func TestParsePlatformClientIDs(t *testing.T) {
}

func TestResolveKeycloakClientID(t *testing.T) {
id, err := resolveKeycloakClientID("ns1", "dep", "", false, "")
defaultTemplate := "spiffe://{{.TrustDomain}}/ns/{{.Namespace}}/sa/{{.ServiceAccount}}"

id, err := resolveKeycloakClientID("ns1", "dep", "", false, "", "")
if err != nil || id != "ns1/dep" {
t.Fatalf("non-spire: %q %v", id, err)
}
_, err = resolveKeycloakClientID("ns1", "dep", "", true, "example.org")
_, err = resolveKeycloakClientID("ns1", "dep", "", true, "example.org", defaultTemplate)
if err == nil {
t.Fatal("expected error for default SA with SPIRE")
}
id, err = resolveKeycloakClientID("ns1", "dep", "mysa", true, "example.org")
id, err = resolveKeycloakClientID("ns1", "dep", "mysa", true, "example.org", defaultTemplate)
if err != nil || id != "spiffe://example.org/ns/ns1/sa/mysa" {
t.Fatalf("spire: %q %v", id, err)
}

// Test custom template
customTemplate := "spiffe://{{.TrustDomain}}/workload/{{.Namespace}}/{{.ServiceAccount}}"
id, err = resolveKeycloakClientID("ns1", "dep", "mysa", true, "example.org", customTemplate)
if err != nil || id != "spiffe://example.org/workload/ns1/mysa" {
t.Fatalf("custom template: %q %v", id, err)
}
}

func clientRegistrationTestScheme(t *testing.T) *runtime.Scheme {
Expand Down
53 changes: 51 additions & 2 deletions kagenti-operator/test/e2e/README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
# E2E Tests

End-to-end tests for the kagenti-operator. The suite runs 8 specs:
End-to-end tests for the kagenti-operator. The suite runs 11 specs:

- **Manager tests** (2 specs) — controller pod readiness and Prometheus metrics
- **AgentCard tests** (6 specs) — webhook validation, auto-discovery, duplicate prevention, audit mode, and SPIRE signature verification
- **ClientRegistration tests** (3 specs) — Keycloak client registration, SPIFFE ID templates, and Secret creation

## Prerequisites

Expand All @@ -20,7 +21,7 @@ The test suite auto-detects Docker vs Podman. No env vars needed.
# Create a fresh Kind cluster
kind delete cluster 2>/dev/null; kind create cluster

# Run all 8 specs (~7 min)
# Run all 11 specs (~9 min)
make test-e2e
```

Expand All @@ -46,6 +47,9 @@ go test ./test/e2e/ -v -ginkgo.v -ginkgo.focus="should reject AgentCard|should n
# Signature verification tests only (~5 min)
go test ./test/e2e/ -v -ginkgo.v -ginkgo.focus="SignatureInvalidAudit|should verify signed"

# ClientRegistration tests only (~3 min)
go test ./test/e2e/ -v -ginkgo.v -ginkgo.focus="ClientRegistration"

# Manager tests only
go test ./test/e2e/ -v -ginkgo.v -ginkgo.focus="Manager"
```
Expand All @@ -66,6 +70,9 @@ kind delete cluster
| Duplicate prevention | Without signature | Webhook rejects a second AgentCard targeting the same workload |
| Audit mode | With signature | Unsigned card syncs (Synced=True) but reports SignatureVerified=False with reason SignatureInvalidAudit |
| Signed agent | With signature | SPIRE-signed card gets SignatureVerified=True, correct SPIFFE ID, Synced=True, and Bound=True |
| Client registration (non-SPIRE) | ClientRegistration | Controller creates client credentials Secret with `namespace/workloadName` format and patches pod template annotation |
| Client registration (SPIRE default) | ClientRegistration | Controller creates SPIFFE ID using default template `spiffe://{domain}/ns/{namespace}/sa/{serviceAccount}` |
| Client registration (SPIRE custom) | ClientRegistration | Controller respects `SPIRE_ID_TEMPLATE` env var for custom SPIFFE ID formats |

## Architecture

Expand Down Expand Up @@ -198,6 +205,48 @@ Test verifies: `SignatureVerified=True` (reason `SignatureValid`),
`signatureSpiffeId = spiffe://example.org/ns/e2e-agentcard-test/sa/signed-agent-sa`,
`Synced=True`, `Bound=True`.

#### Client registration (non-SPIRE mode)

Controller is patched with `--enable-operator-client-registration=true --spire-trust-domain=example.org`.
Deploys a mock Keycloak server, `authbridge-config` ConfigMap with `SPIRE_ENABLED=false`, and
`keycloak-admin-secret`. Deploys `clientreg-agent` workload.

1. Controller watches the Deployment (has `kagenti.io/type=agent` label)
2. Reads `authbridge-config` and `keycloak-admin-secret` from workload namespace
3. Computes client ID as `namespace/workloadName` (non-SPIRE format)
4. Registers client with mock Keycloak via admin API
5. Creates Secret with `client-id.txt` and `client-secret.txt` (owner: Deployment)
6. Patches Deployment pod template with `kagenti.io/keycloak-client-credentials-secret-name` annotation

Test verifies: Secret created, contains both keys, client ID format is `e2e-clientreg-test/clientreg-agent`,
and pod template annotation references the Secret name.

#### Client registration (SPIRE mode with default template)

Updates `authbridge-config` to `SPIRE_ENABLED=true`. Deploys `clientreg-spire-agent` with dedicated
ServiceAccount `clientreg-spire-sa`.

1. Controller detects SPIRE is enabled
2. Validates ServiceAccount is not `default`
3. Renders default SPIFFE ID template: `spiffe://{{.TrustDomain}}/ns/{{.Namespace}}/sa/{{.ServiceAccount}}`
4. Computes client ID as `spiffe://example.org/ns/e2e-clientreg-test/sa/clientreg-spire-sa`
5. Registers client and creates Secret with SPIFFE ID as client ID

Test verifies: Client ID matches expected SPIFFE format with default template.

#### Client registration (SPIRE mode with custom template)

Patches operator Deployment to set env var `SPIRE_ID_TEMPLATE=spiffe://{{.TrustDomain}}/workload/{{.Namespace}}/{{.ServiceAccount}}`.
Waits for operator restart. Deploys `clientreg-custom-agent` with ServiceAccount `clientreg-custom-sa`.

1. Controller reads custom template from environment variable
2. Renders custom template with workload metadata
3. Computes client ID as `spiffe://example.org/workload/e2e-clientreg-test/clientreg-custom-sa`
4. Registers client and creates Secret

Test verifies: Client ID matches expected SPIFFE format with custom template. Restores operator
to default configuration after test.

## Troubleshooting

**Stale cluster state** — if you see errors about namespaces being terminated or cert-manager TLS failures, delete and recreate the cluster:
Expand Down
Loading
Loading