🤖 feat: add CoderProvisioner CRD and controller for external provisioner daemons

[ThomasK33]

Summary

Adds a new CoderProvisioner CRD and controller that deploys external provisioner daemons for a Coder control plane. The controller:

1. References a CoderControlPlane CR to discover the coderd URL
2. Creates and manages provisioner keys in coderd via the API
3. Stores key material in Kubernetes Secrets
4. Reconciles a Deployment running provisionerd start with the correct environment
5. Provisions RBAC (ServiceAccount, Role, RoleBinding) for the provisioner pods
6. Cleans up remote provisioner keys on CR deletion via a finalizer

Background

Coder supports running provisioner daemons outside of coderd (the CLI subcommand provisionerd start). This lets operators scale workspace build/apply concurrency independently of the main Coder control-plane. This feature bridges the coder-k8s operator with external provisioner management.

Implementation

New files

• api/v1alpha1/coderprovisioner_types.go — CRD spec/status types with bootstrap, key management, and pod template customization fields
• internal/coderbootstrap/provisionerkeys.go — EnsureProvisionerKey and DeleteProvisionerKey methods using vendored codersdk
• internal/coderbootstrap/provisionerkeys_test.go — httptest-based unit tests for provisioner key operations
• internal/controller/coderprovisioner_controller.go — full reconciler with finalizer-based deletion, key rotation support, and child resource management
• internal/controller/coderprovisioner_controller_test.go — comprehensive envtest-based controller tests (6 test cases)
• config/samples/coder_v1alpha1_coderprovisioner.yaml — sample CR manifest

Modified files

• internal/coderbootstrap/client.go — extended Client interface with provisioner key methods
• internal/app/controllerapp/controllerapp.go — wired new controller into manager
• internal/controller/workspaceproxy_controller_test.go — updated fake bootstrap client for interface compliance

Generated artifacts

• api/v1alpha1/zz_generated.deepcopy.go — deepcopy methods for new types
• config/crd/bases/coder.com_coderprovisioners.yaml — CRD manifest
• config/rbac/role.yaml — RBAC rules for new resources
• docs/reference/api/coderprovisioner.md — auto-generated API reference
• mkdocs.yml — nav entry for new API doc

Validation

• make build ✅
• make test ✅
• make test-integration ✅
• make lint ✅
• make verify-vendor ✅
• make docs-reference-check ✅

Risks

• The EnsureProvisionerKey implementation does not yet support tag-change rotation (existing keys with mismatched tags are not automatically deleted/recreated). This is acceptable for the first iteration since tag changes would require manual key rotation or CR deletion/recreation.
• Role/RoleBinding resources are reconciled with owner refs but SetupWithManager only watches Deployment/Secret/ServiceAccount (not Role/RoleBinding). Changes to Role/RoleBinding won't trigger reconciliation unless the CR itself is modified.

---

📋 Implementation Plan

Plan: Managed external provisioner daemons (CoderProvisioner)

Context / Why

Coder supports running provisioner daemons outside of coderd (the CLI subcommand provisionerd start). This lets us scale workspace build/apply concurrency independently of the main Coder control-plane.

Goal for coder-k8s: add a new CRD + controller reconciler that can:

1. Deploy one-or-more external provisioner daemon Pods in Kubernetes.
2. (Option B) Automatically create and manage a Coder provisioner key in coderd.
3. Store that key in a Kubernetes Secret and wire it into the provisioner Deployment so the daemons automatically join the referenced coderd instance.
4. Clean up remote keys on deletion (finalizer), and rotate keys when needed.

Non-goals (first iteration): autoscaling (HPA), multi-namespace workspace RBAC fan-out, observing per-daemon connection health via coderd APIs.

---

Evidence (what was inspected)

coder/coder (external provisioners)

• enterprise/cli/provisionerdaemonstart.go: coder provisionerd start flags/env for CODER_URL, CODER_PROVISIONER_DAEMON_KEY, org, tags.
• enterprise/coderd/provisionerdaemons.go: coderd WebSocket endpoint /api/v2/organizations/{org}/provisionerdaemons/serve.
• coderd/httpmw/provisionerdaemon.go: auth headers (Coder-Provisioner-Daemon-Key, PSK, session token).
• provisionerd/proto/provisionerd.proto: DRPC service (AcquireJobWithCancel, UpdateJob, CompleteJob, …).
• Helm patterns:
  • helm/provisioner/values.yaml, helm/provisioner/templates/_coder.tpl: args: [provisionerd, start], secretKeyRef for CODER_PROVISIONER_DAEMON_KEY.
  • helm/libcoder/templates/_helpers.tpl: RBAC rules (pods + PVCs, optionally deployments).

coder-k8s (current patterns)

• api/v1alpha1/codercontrolplane_types.go: CoderControlPlaneStatus.URL (in-cluster coderd URL), status pattern (Phase, Conditions).
• api/v1alpha1/types_shared.go: SecretKeySelector helper type.
• internal/controller/workspaceproxy_controller.go: secret generation pattern (ensureTokenSecret), CreateOrUpdate, owner refs.
• internal/coderbootstrap/client.go: coderd API abstraction (currently EnsureWorkspaceProxy).
• Vendored SDK: vendor/github.com/coder/coder/v2/codersdk/provisionerdaemons.go contains provisioner key methods:
  • CreateProvisionerKey, ListProvisionerKeys, GetProvisionerKey, DeleteProvisionerKey.

---

Implementation details

1) API: add CoderProvisioner CRD (v1alpha1)

Create a new types file:

• api/v1alpha1/coderprovisioner_types.go

Key design:

• Reference a CoderControlPlane by name (same namespace) and derive CODER_URL from its status.
• Require a bootstrap session token (user-provided) so the operator can call coderd APIs to create/delete provisioner keys.
• Store the created provisioner key material only in a Kubernetes Secret (never in CR status).

Proposed spec/status (shape; exact names can be tuned to match repo conventions):

// CoderProvisionerSpec defines desired state.
type CoderProvisionerSpec struct {
    // ControlPlaneRef identifies which coderd instance to join.
    ControlPlaneRef corev1.LocalObjectReference `json:"controlPlaneRef"`

    // OrganizationName defaults to "default".
    OrganizationName string `json:"organizationName,omitempty"`

    // Bootstrap is required to mint/rotate provisioner keys.
    Bootstrap CoderProvisionerBootstrapSpec `json:"bootstrap"`

    // Key management options.
    Key CoderProvisionerKeySpec `json:"key,omitempty"`

    Replicas *int32 `json:"replicas,omitempty"`

    // Tags to attach to the provisioner key (job routing).
    Tags map[string]string `json:"tags,omitempty"`

    // WorkspacePermissions controls Role/RoleBinding creation for the provisioner SA.
    WorkspacePermissions WorkspacePermissionsSpec `json:"workspacePermissions,omitempty"`

    // PodTemplate customizations.
    Image            string          `json:"image,omitempty"` // default: ControlPlane.spec.image
    ExtraEnv         []corev1.EnvVar `json:"extraEnv,omitempty"`
    ExtraArgs        []string        `json:"extraArgs,omitempty"` // appended after `provisionerd start`
    Resources        corev1.ResourceRequirements `json:"resources,omitempty"`

    TerminationGracePeriodSeconds *int64 `json:"terminationGracePeriodSeconds,omitempty"`
}

type CoderProvisionerBootstrapSpec struct {
    // Session token with permission to manage provisioner keys.
    CredentialsSecretRef SecretKeySelector `json:"credentialsSecretRef"`
}

type CoderProvisionerKeySpec struct {
    // Name in coderd; default derived from the CR name.
    Name string `json:"name,omitempty"`

    // SecretName to store the plaintext key; default derived from CR name.
    SecretName string `json:"secretName,omitempty"`

    // SecretKey is the data key; default "key" (to match upstream Helm chart).
    SecretKey string `json:"secretKey,omitempty"`
}

type CoderProvisionerStatus struct {
    ObservedGeneration int64 `json:"observedGeneration,omitempty"`
    ReadyReplicas      int32 `json:"readyReplicas,omitempty"`
    Phase              string `json:"phase,omitempty"`
    Conditions         []metav1.Condition `json:"conditions,omitempty"`

    // Remote metadata to support cleanup/rotation decisions.
    OrganizationID string `json:"organizationID,omitempty"` // uuid
    ProvisionerKeyID string `json:"provisionerKeyID,omitempty"` // uuid
    ProvisionerKeyName string `json:"provisionerKeyName,omitempty"`
    SecretRef SecretKeySelector `json:"secretRef,omitempty"`
}

Notes:

• Use existing SecretKeySelector type from api/v1alpha1/types_shared.go.
• Prefer deterministic key naming so finalizer cleanup can always delete by name.
• Add a finalizer constant (e.g. coder.com/provisioner-key-cleanup).

2) coderd API integration: extend internal/coderbootstrap

Add provisioner-key operations so controllers don’t embed raw codersdk logic:

Files:

• internal/coderbootstrap/client.go (extend interface)
• internal/coderbootstrap/provisionerkeys.go (new)

Add a method that supports rotation when key material is required:

type EnsureProvisionerKeyRequest struct {
    CoderURL     string
    SessionToken string

    OrganizationName string
    KeyName          string
    Tags             map[string]string

    // If true, ensure response includes plaintext key material (rotate if necessary).
    KeyMaterialRequired bool
}

type EnsureProvisionerKeyResponse struct {
    OrganizationID uuid.UUID
    KeyID          uuid.UUID
    KeyName        string

    // Non-empty only if created/rotated.
    Key string
}

type Client interface {
    EnsureWorkspaceProxy(ctx context.Context, req EnsureWorkspaceProxyRequest) (EnsureWorkspaceProxyResponse, error)
    EnsureProvisionerKey(ctx context.Context, req EnsureProvisionerKeyRequest) (EnsureProvisionerKeyResponse, error)
    DeleteProvisionerKey(ctx context.Context, coderURL, sessionToken, organizationName, keyName string) error
}

Implementation approach (using vendored codersdk):

1. client := codersdk.New(req.CoderURL); client.SetSessionToken(req.SessionToken).
2. Resolve org ID from org name (use codersdk org endpoints; cache not required initially).
3. ListProvisionerKeys(orgID); find by KeyName.
4. If missing → CreateProvisionerKey(orgID, {Name: KeyName, Tags: req.Tags}) and return plaintext key.
5. If exists but tags differ OR KeyMaterialRequired==true and we don’t want to rely on unknown existing secret state:
   • Delete by name DeleteProvisionerKey(orgID, KeyName) (or delete+recreate if create fails due to conflict)
   • Recreate → return new plaintext key.
6. Else return metadata with Key=="".

Defensive programming:

• Treat “secret missing” as a signal to rotate because coderd only returns plaintext key once.
• When a secret does contain a key, optionally validate it with GetProvisionerKey(ctx, keyMaterial) and ensure the returned Name/Tags/OrgID match expectations; if not, rotate.

3) Controller: CoderProvisionerReconciler

New file:

• internal/controller/coderprovisioner_controller.go

RBAC markers to add (in this new controller file):

// +kubebuilder:rbac:groups=coder.com,resources=coderprovisioners,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=coder.com,resources=coderprovisioners/status,verbs=get;update;patch
// +kubebuilder:rbac:groups="",resources=secrets;serviceaccounts,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=rbac.authorization.k8s.io,resources=roles;rolebindings,verbs=get;list;watch;create;update;patch;delete

Reconcile flow (high level):

flowchart TD
  A[Fetch CoderProvisioner] --> B{DeletionTimestamp?}
  B -->|yes| Z[Finalizer: delete coderd provisioner key; remove finalizer]
  B -->|no| C[Fetch referenced CoderControlPlane]
  C --> D[Get coderd URL from controlPlane.status.url]
  D --> E[Read bootstrap session token secret]
  E --> F[Ensure/Rotate provisioner key via coderbootstrap]
  F --> G[Upsert K8s Secret with key material]
  G --> H[Upsert ServiceAccount]
  H --> I[Upsert Role/RoleBinding if workspacePermissions enabled]
  I --> J[Upsert Deployment: args provisionerd start; env CODER_URL + CODER_PROVISIONER_DAEMON_KEY]
  J --> K[Update Status: phase, readyReplicas, key metadata]

Loading

Concrete resource behavior:

Secret

• Name: spec.key.secretName default ${crName}-provisioner-key.
• Data key: spec.key.secretKey default "key".
• Annotations: store key name + key ID + org ID (helps recovery if status wiped).
• OwnerRef: CoderProvisioner.

Deployment

• Image: spec.image default controlPlane.spec.image.
• Args: provisionerd start plus spec.extraArgs.
• Env:
  • CODER_URL = controlPlane.status.url
  • CODER_ORGANIZATION = spec.organizationName (if set)
  • CODER_PROVISIONER_DAEMON_KEY from SecretKeyRef
  • optional: CODER_PROVISIONER_DAEMON_NAME from downward API (metadata.name) for observability
• Set terminationGracePeriodSeconds default 600.
• Pod template annotation checksum/secret computed from key Secret data to force rollout on rotations.

ServiceAccount + Role/RoleBinding

• Create SA (default name derived from CR) and bind a Role with rules copied from upstream Helm:
  • core: pods, persistentvolumeclaims (CRUD+watch)
  • optional: apps/deployments (CRUD+watch)

Finalizer behavior:

• On delete: call coderbootstrap.DeleteProvisionerKey(...) (by name) then remove finalizer.
• Child k8s resources should be GC’d via OwnerRefs.

4) Wire into manager/app

• Add scheme registration for the new API type (usually automatic via init() in the types file, but verify api/v1alpha1/groupversion_info.go registration).
• Update internal/app/controllerapp/controllerapp.go to:
  • construct the shared coderbootstrap.SDKClient (already done for WorkspaceProxy)
  • register CoderProvisionerReconciler with that client.

5) Tests

Fast/default tests (no real coderd):

• Controller envtest coverage similar to workspaceproxy_controller_test.go:

  • internal/controller/coderprovisioner_controller_test.go
    • Create CoderControlPlane with status.url populated.
    • Create bootstrap credentials Secret.
    • Create CoderProvisioner CR.
    • Inject a fake coderbootstrap.Client into the reconciler to simulate:
      • initial key create returning {Key: "plaintext", KeyID: ...}
      • tag change causing rotation
      • deletion finalizer calling delete
    • Assert:
      • Secret created with expected key data
      • Deployment created with expected args/env/ownerrefs
      • SA/Role/RoleBinding created when enabled
      • Status fields updated

• Unit tests for coderbootstrap.EnsureProvisionerKey:

  • Prefer an httptest.Server that implements the minimal coderd endpoints used:
    • POST/GET/DELETE /api/v2/organizations/{org_id}/provisionerkeys
    • (optional) GET /api/v2/provisionerkeys/{key} for validating key material
  • This keeps tests deterministic and avoids requiring Docker/Postgres.

Optional end-to-end (real enterprise coderd + real provisionerd join):

Note: envtest does not run Pods, so this can validate the API interactions + that the produced Secret/key actually allows a real provisionerd client to connect to coderd. To validate the Kubernetes Deployment actually runs and joins, we’d need a Kind/e2e suite.

• Add a build-tagged integration test (e.g. //go:build integration) that:
  1. Starts an Enterprise coderd instance via github.com/coder/coder/v2/enterprise/coderd/coderdenttest with AllFeatures: true (enables external_provisioner_daemons).
     • Requires Docker because coderd’s test harness uses dbtestutil.NewDB() (dockertest-managed Postgres) by default.
     • Will increase vendoring (pulls in enterprise/ test helpers + deps).
  2. Runs the CoderProvisionerReconciler against that coderd (real coderbootstrap.SDKClient, not fake).
  3. After reconcile, reads the generated K8s Secret and starts an in-process provisionerd.Server using codersdk.Client.ServeProvisionerDaemon with ProvisionerKey set to the secret value.
  4. Asserts coderd sees the daemon (via client.OrganizationProvisionerDaemons / client.ProvisionerDaemons).

6) Generated artifacts and samples

• Run make codegen (deepcopy updates).
• Run make manifests (CRD + RBAC role updates).
• Run make docs-reference (updates docs/reference/api/* for the new CRD types).
• Update/extend docs explaining how to:
  • create the bootstrap session token Secret
  • deploy a CoderProvisioner and what tags mean for job routing
• Add config/samples/coder_v1alpha1_coderprovisioner.yaml demonstrating:
  • bootstrap.credentialsSecretRef
  • controlPlaneRef
  • tags and replicas

---

Validation checklist (when implementing)

• make test
• make build
• make lint
• make verify-vendor (only if deps changed)
• make codegen
• make manifests
• make docs-check

---

Generated with mux • Model: anthropic:claude-opus-4-6 • Thinking: xhigh • Cost: $1.78

[ThomasK33]

@codex review

[ThomasK33]

@codex review

[ThomasK33]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b2c135b1e7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

@codex review

Addressed both P1 issues:

1. Best-effort finalizer cleanup: Deletion now handles missing CoderControlPlane or bootstrap credentials gracefully (logs warning, proceeds to finalizer removal) instead of blocking the CR in Terminating state.

2. Key rotation on secret loss: When the provisioner key secret is deleted but the key already exists in coderd (which won't return plaintext again), the controller now rotates the key by deleting and recreating it to obtain fresh material.

Added test coverage for the control-plane-deleted-first scenario.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 01cd967b11

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

@codex review

Addressed the remaining P1 and P2 issues:

1. P1 - Skip remote cleanup when control plane URL is unavailable: All fetchControlPlane errors during deletion are now treated as non-blocking (not just NotFound). This handles the case where the control plane exists but has an empty status.url.

2. P2 - Use persisted key identity for finalizer cleanup: Finalizer cleanup now uses status.ProvisionerKeyName (reflecting what was actually created in coderd) instead of the current spec value. Falls back to spec-derived name only if status is empty.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 811a212c52

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

@codex review

Addressed both remaining issues:

1. P1 - All token-read errors are now best-effort during deletion: Any failure reading bootstrap credentials (not just NotFound) is logged and skipped, preventing stuck Terminating states from misconfigured or missing credentials.

2. P2 - Use persisted org identity: Finalizer cleanup now prefers status.OrganizationID (the resolved value from when the key was created) over the current spec value. Falls back to spec-derived name only if status is empty.

The deletion cleanup is now fully best-effort: any prerequisite failure (control plane, URL, credentials) is logged and the finalizer is removed regardless.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f7dc3d9675

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

@codex review

Addressed both issues:

1. P1 - Best-effort key deletion: DeleteProvisionerKey failures during deletion are now logged and skipped. The entire deletion path is now fully best-effort — no step blocks finalizer removal.

2. P2 - Validate secret data key: Secret usability check now verifies both object existence AND non-empty value at the configured data key. A secret with missing/empty key entry triggers key rotation/recovery.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6c4afaf89e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

@codex review

Added Owns(&rbacv1.Role{}) and Owns(&rbacv1.RoleBinding{}) to SetupWithManager so RBAC drift is self-healed promptly.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c0ed9d158d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

@codex review

Fixed both issues:

1. P1: Deletion now uses spec-derived org name (not status UUID) for DeleteProvisionerKey.
2. P2: provisionerServiceAccountName now uses FNV hash truncation for names > 63 chars.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8cffa8ac73

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

@codex review

[ThomasK33]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. More of your lovely PRs please.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ThomasK33]

https://mux.md/Rck7l#Cr9ZukWjvxIcRA