Skip to content

feat(controller): engine-agnostic CA extraction for the tenant trust anchor - #3299

Closed
Aleksei Sviridkin (lexfrei) wants to merge 38 commits into
mainfrom
feat/ca-extraction-controller
Closed

feat(controller): engine-agnostic CA extraction for the tenant trust anchor#3299
Aleksei Sviridkin (lexfrei) wants to merge 38 commits into
mainfrom
feat/ca-extraction-controller

Conversation

@lexfrei

@lexfrei Aleksei Sviridkin (lexfrei) commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

What this PR does

A tenant that connects to a managed TLS endpoint needs the CA certificate (ca.crt) to verify the server, and nothing more. Today the platform delivers it for no engine except kafka, and the in-flight per-app TLS work would each hand the tenant a Secret that also carries a private key — the cert-manager CA Secret holds tls.key, CloudNativePG's <release>-ca holds ca.key. Granting read on one of those to deliver the trust anchor hands over key material too.

This adds a controller that publishes, for every engine, one canonical key-free object per release: an Opaque Secret named <release>.tenant-ca holding only ca.crt.

The dot in that name is load-bearing, and it is the third name this object has had. <release>-ca-cert was rejected because Percona Server for MongoDB creates a Secret of exactly that name and puts a private key in it. Its replacement <release>-tenant-ca was rejected too, for a subtler reason: it collides across releases. For an application foo the projection would be postgres-foo-tenant-ca, and for a sibling application foo-tenant CloudNativePG's own CA is postgres-foo-tenant + -ca — the same object. One direction of that collision cannot be guarded: if the projection is written first, CNPG rejects the key-free Secret (missing ca.key secret data) and the sibling's PKI never completes, so that application never starts. <release>.tenant-ca is disjoint by character class instead of by survey — application names are DNS-1035 labels and structurally cannot contain a dot, release prefixes are dot-free, and Secret names are DNS-1123 subdomains where a dot is legal, so no prefix + application + operator suffix can ever produce a dotted name, for any operator, now or later.

Source discovery has two legs behind one write path, and the declared leg wins. When an ApplicationDefinition declares spec.caCert, that named source is authoritative and a labelled Secret is not consulted for the engine; the label leg serves only engines with no declaration. This ordering is a security boundary: the declaration lives on a platform object no tenant can write, while a publish label sits on a Secret a namespace writer could create — so a declared engine's trust anchor cannot be swapped for a forged one.

The name-driven (declared) leg exists because CloudNativePG and Percona Server for MongoDB create their CA Secret themselves and cannot be made to label it. The label-driven leg serves the cert-manager-minting charts, which opt in through Certificate.spec.secretTemplate.labels; on it a required label internal.cozystack.io/publish-ca-cert-release names the release, because a cert-manager-issued Secret carries no OwnerReference (the platform ships enableCertificateOwnerRef: false) and no Helm metadata, so nothing else can attribute it to a release.

The write path is the single place bytes reach a projection, on both legs. It copies exactly one whitelisted key under the canonical ca.crt, and it parses rather than pattern-matches: every PEM block must decode and be accepted by x509.ParseCertificate, with no trailing remainder and no private-key block anywhere. A header check is not sufficient — PEM armour around arbitrary bytes satisfies it, and pem.Decode does not close the gap either, because it validates the armour and the base64 rather than the contents.

The projection is owner-referenced to the application's HelmRelease with BlockOwnerDeletion: false (the apps.cozystack.io kinds are virtual, so there is no application CR in etcd to reference), and the controller requires sole ownership so a second owner reference cannot keep a retired anchor alive after the application is gone. A Secret at the canonical name the controller did not create is never overwritten; the collision surfaces as a Warning Event.

Withdrawal distinguishes a removed declaration from a source that is merely absent: deleting a CA Secret is how a cert-manager reissue is forced, so absence holds the last good anchor and waits, while a declaration that is gone withdraws the anchor definitively. The projection records which leg produced it, because that fact cannot be re-derived once the declaration is gone.

The label leg is guarded at admission. A ValidatingAdmissionPolicy restricts writes to any Secret carrying internal.cozystack.io/publish-ca-cert to cert-manager's controller ServiceAccount, on both CREATE and UPDATE (checking the old object too, so relabel-away is also caught), and a bats check renders the shipped cert-manager package and asserts the pinned identity still matches, so a repackage cannot silently break the seam.

postgres is wired as the first consumer, validating the mechanism against the hardest input — an operator-created, asynchronous, key-bearing CA Secret on the name-driven leg. The anchor reaches tenants through core.cozystack.io/tenantsecrets, gated on the lineage webhook's verdict rather than a name match, and the chainsaw test asserts the tenant-facing path end to end, including the negative that CNPG's key-bearing CA must not resolve. The remaining engines are tracked in #2814.

Known limitation: a tenant who strips the internal.cozystack.io/ca-cert-copy marker from their own projection makes the controller treat it as foreign and stop updating that one anchor — self-inflicted, same-namespace, no key leak, a direct consequence of never adopting a Secret it cannot prove it owns.

Release note

feat(controller): add an engine-agnostic controller that publishes each application's CA certificate as a key-free `<release>.tenant-ca` Secret, so a tenant can verify a managed TLS endpoint without gaining access to private key material; postgres is wired as the first consumer

Closes #3286

Summary by CodeRabbit

  • New Features
    • Added spec.application.caCert to source the CA trust anchor from sourceSecretName and optional sourceKey (default ca.crt).
    • CA is projected into a canonical, key-free tenant Secret: <release>.tenant-ca (contains only ca.crt).
  • Security / Policy
    • Introduced fail-closed admission control and hardened CA publishing/projection to prevent private-key exposure and smuggled PEM content.
    • Updated spec.release.prefix validation pattern for consistent dot-free naming.
  • Documentation
    • Updated PostgreSQL TLS retrieval instructions to use <release>.tenant-ca.
  • Tests
    • Added controller, policy, RBAC, and end-to-end coverage for CA wiring, validation, ownership, and safe pruning.

@github-actions github-actions Bot added area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review kind/feature Categorizes issue or PR as related to a new feature labels Jul 15, 2026
@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Adds an ApplicationDefinition CA source contract, a controller that creates key-free <release>.tenant-ca Secrets, dedicated cache wiring, admission controls, and Postgres/TLS integration with unit, end-to-end, and policy validation.

Changes

CA certificate extraction

Layer / File(s) Summary
ApplicationDefinition CA contract
api/v1alpha1/..., packages/system/application-definition-crd/...
Adds spec.application.caCert with a required source Secret name, default ca.crt key, validation, deepcopy support, and CRD schema.
Source resolution and tenant projection
internal/controller/cacert/reconciler.go
Resolves declared or labelled sources, validates certificate data, projects only ca.crt into <release>.tenant-ca, and handles ownership, collisions, retries, rotation, and pruning.
Controller watches and runtime registration
internal/controller/cacert/..., cmd/cozystack-controller/main.go
Adds scoped Secret caching, source and definition event mapping, and dedicated cache-cluster registration.
Admission and Secret permissions
packages/system/cozystack-basics/..., packages/system/cozystack-controller/...
Adds fail-closed admission for labelled CA-source writes and validates Secret permissions and writer identity.
Tenant CA naming and application wiring
packages/library/cozy-lib/..., packages/system/postgres-rd/..., packages/apps/postgres/..., packages/tests/cozy-lib-tests/...
Uses the tenant CA name and labels in the TLS helper, documentation, RBAC checks, and Postgres configuration.
Controller behavior coverage
internal/controller/cacert/reconciler_test.go
Tests source selection, sanitization, security checks, lifecycle behavior, collisions, retries, pruning, watches, and cache scoping.
Postgres and platform validation
internal/controller/cacert/postgres_wiring_test.go, hack/e2e-chainsaw/postgres/..., hack/e2e-chainsaw/cacert/...
Verifies projection behavior, tenant-secret exposure, source-key preservation, and admission-policy enforcement.
Supporting updates
.gitignore, hack/select-install.sh, hack/e2e-chainsaw/README.md
Updates build-artifact ignores and registers the new Chainsaw suite.

Estimated code review effort: 5 (Critical) | ~90+ minutes

Sequence Diagram(s)

sequenceDiagram
  participant HelmRelease
  participant CACertReconciler
  participant ApplicationDefinition
  participant SourceSecret
  participant TenantCASecret
  HelmRelease->>CACertReconciler: reconcile application release
  CACertReconciler->>ApplicationDefinition: read caCert source declaration
  CACertReconciler->>SourceSecret: read declared or labelled CA data
  CACertReconciler->>CACertReconciler: validate PEM and retain ca.crt
  CACertReconciler->>TenantCASecret: create or update key-free projection
Loading

Possibly related issues

Possibly related PRs

Suggested labels: kind/api-change, area/testing

Suggested reviewers: kvaps

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Out of Scope Changes check ⚠️ Warning The .gitignore update adds root-level Go binary ignores that are unrelated to the CA extraction feature. Remove the .gitignore cleanup from this PR or split it into a separate housekeeping change.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title is clear, concise, and matches the main controller-focused CA extraction change.
Linked Issues check ✅ Passed The PR adds the extraction controller, source selection, sanitization, ownership, admission guard, and tenant projection required by #3286.
Docstring Coverage ✅ Passed Docstring coverage is 87.50% which is sufficient. The required threshold is 80.00%.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/ca-extraction-controller

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the size/XXL This PR changes 1000+ lines, ignoring generated files label Jul 15, 2026
@lexfrei
Aleksei Sviridkin (lexfrei) marked this pull request as ready for review July 15, 2026 01:51
@dosubot dosubot Bot added the area/platform Issues or PRs related to platform infrastructure (bundle, flux, talos, installer) label Jul 15, 2026
@lexfrei Aleksei Sviridkin (lexfrei) removed the area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review label Jul 15, 2026
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request implements a secure, engine-agnostic framework for distributing CA certificates to tenants. By decoupling the trust anchor from the underlying operator-managed secrets, the platform prevents the accidental exposure of private key material. The solution provides a robust, fail-closed extraction path that supports both label-based and explicitly declared sources, ensuring consistent and secure trust anchor delivery across various managed services.

Highlights

  • New CA-Extraction Controller: Introduced a new controller that extracts CA certificates into key-free Opaque Secrets named -tenant-ca, ensuring tenants can verify TLS endpoints without accessing private keys.
  • Dual-Leg Source Discovery: Implemented two discovery methods: a label-driven approach for cert-manager-minted secrets and a name-driven (declared) approach for operator-created secrets that cannot be labeled.
  • Security Hardening: Added write-path sanitization to block private key material and implemented a ValidatingAdmissionPolicy to restrict source secret modifications to authorized service accounts.
  • Postgres Integration: Wired the Postgres engine as the first consumer of this mechanism, validating the extraction logic against complex, key-bearing operator secrets.
New Features

🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Ignored Files
  • Ignored by pattern: **/zz_generated.*.go (1)
    • api/v1alpha1/zz_generated.deepcopy.go
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment Gemini (@gemini-code-assist) Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on Gemini (@gemini-code-assist) comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a CA-extraction controller (cacert) that projects the TLS trust anchor of managed applications into a canonical, key-free Secret (<release>-tenant-ca) in the release namespace, allowing tenants to securely read CA certificates without exposing private keys. It updates the ApplicationDefinition CRD and CRD schema to support authoritative caCert declarations, configures the Postgres application definition to use this new mechanism, and updates the cozy-lib library chart helper to align with the new canonical naming and labelling. Additionally, it introduces a ValidatingAdmissionPolicy to restrict writes to CA-source Secrets to cert-manager only, updates controller RBAC permissions, and adds comprehensive unit and integration tests. I have no feedback to provide as there are no review comments to assess.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/controller/cacert/reconciler.go`:
- Around line 565-576: Treat any source Secret already occupying the canonical
projection name as a collision, regardless of whether it contains private key
material: update reconcileSourceAtCanonicalName to emit the required collision
warning and requeue instead of silently succeeding. Update
internal/controller/cacert/reconciler_test.go lines 524-550 to assert the
collision warning and resulting behavior rather than a successful no-op.
- Around line 885-895: Validate the namespace parsed from SourceRefAnnotation
against hr.Namespace before calling r.Reader.Get in the existing source-lookup
block. Reject or ignore references where ns differs from hr.Namespace, ensuring
no cross-namespace Secret read occurs while preserving same-namespace lookup
behavior.
- Around line 495-502: Update the reconciliation path around the src == nil
branch and pruneProjection so the persisted source mode remains available when
the declared Secret disappears, and pass the current ApplicationDefinition into
pruning to distinguish a removed declaration from a temporarily absent source.
Ensure removing spec.caCert withdraws the previously projected trust anchor, and
add a regression test covering source disappearance before declaration removal.
- Around line 960-968: Update projectionData to fully decode and parse every PEM
block as a certificate, rejecting malformed or truncated payloads, trailing
bytes, and non-certificate blocks while preserving private-key rejection. In
internal/controller/cacert/reconciler.go lines 960-968, replace header matching
with complete validation; in internal/controller/cacert/reconciler_test.go lines
66-67, use valid certificate fixtures; and in
internal/controller/cacert/reconciler_test.go lines 1383-1398, add malformed,
trailing-data, and non-certificate-block cases.
- Around line 1085-1089: Update the rendered Secret name validation after
strings.TrimSpace in the surrounding name-rendering function to reject values
that are not valid DNS1123 subdomains, using the existing
CACertDeclarationInvalid error/result convention instead of the generic error.
Preserve the empty-name check and return the validated name only when it
satisfies the Kubernetes naming rules.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 8f836b39-3c2b-4df5-a46a-93f85ee43477

📥 Commits

Reviewing files that changed from the base of the PR and between 6f58898 and c5d95dc.

📒 Files selected for processing (16)
  • api/v1alpha1/applicationdefinitions_types.go
  • api/v1alpha1/zz_generated.deepcopy.go
  • cmd/cozystack-controller/main.go
  • internal/controller/cacert/postgres_wiring_test.go
  • internal/controller/cacert/reconciler.go
  • internal/controller/cacert/reconciler_test.go
  • packages/library/cozy-lib/templates/_tls.tpl
  • packages/system/application-definition-crd/definition/cozystack.io_applicationdefinitions.yaml
  • packages/system/cozystack-basics/templates/publish-ca-cert-writer-policy.yaml
  • packages/system/cozystack-basics/tests/publish-ca-cert-writer-policy_test.yaml
  • packages/system/cozystack-controller/Makefile
  • packages/system/cozystack-controller/templates/rbac.yaml
  • packages/system/cozystack-controller/tests/rbac_test.yaml
  • packages/system/postgres-rd/cozyrds/postgres.yaml
  • packages/tests/cozy-lib-tests/templates/tests/tls-cacert.yaml
  • packages/tests/cozy-lib-tests/tests/tls_cacert_test.yaml

Comment thread internal/controller/cacert/reconciler.go Outdated
Comment thread internal/controller/cacert/reconciler.go Outdated
Comment thread internal/controller/cacert/reconciler.go Outdated
Comment thread internal/controller/cacert/reconciler.go Outdated
Comment thread internal/controller/cacert/reconciler.go Outdated

@myasnikovdaniil myasnikovdaniil left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the whole controller, the VAP, the RBAC delta, the cozy-lib guard, and the tests as the tree rather than the diff, and traced both source legs end to end against the tenant read path. The security story holds: tenants read trust anchors only through the label-gated, read-only tenantsecrets API and cannot write raw Secrets, the label leg is admission-gated to cert-manager, and the declared leg stays authoritative over the forgeable label leg — so the change fails closed on every tenant-influenceable path, is namespace-confined with no cross-tenant vector, and is unusually well tested (forged-label override, forged-projection healing, non-Opaque collision, recreate-rehome, poisoned rotation, prune discrimination).

Approving — everything below is non-blocking. The two follow-ups most worth doing are the x509-parse hardening of the write-path guard (it currently validates by header substring only) and tightening the prune/canonical-name corner, where a retired anchor can linger if a declaration is removed while its source is momentarily absent. Two nits I did not inline: selectorsDigest keeps only the low 64 bits of the SHA-256 (reconciler.go:952), and applicationDefinition returns the first kind match without flagging duplicate kinds (reconciler.go:696) — both harmless with platform-authored input.

if containsPrivateKey(pem) {
return nil, errPrivateKey
}
if !certificateHeader.MatchString(pem) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking, defense-in-depth. This guard validates the trust anchor by header substring — BEGIN CERTIFICATE present and no BEGIN … PRIVATE KEY — but never decodes the PEM or runs x509.ParseCertificate, so certificate armor wrapped around non-certificate bytes (e.g. headerless DER) passes and is copied verbatim. It is not reachable from a shipped path: both legs' sources come from trusted writers (the operator-created <release>-ca, or cert-manager, which the writer VAP gates) and a tenant can write neither. But since this is the stated fail-closed boundary, worth making the check match the claim: decode with encoding/pem and assert every block is a CERTIFICATE that x509.ParseCertificate accepts, with no trailing remainder. The mirrored chart guard in _tls.tpl shares the limitation.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Go guard does decode: certificateChainPEM runs pem.Decode + x509.ParseCertificate on every block and rejects any trailing remainder, and 15b7442 goes further — the projection is rebuilt from the re-encoded parsed DER, so any byte the guard did not parse as a certificate (headerless DER, a JWK, preamble) is structurally unreachable rather than copied. The chart mirror in _tls.tpl is anchored to the whole value (bff1e3f), which is the most a Helm template can enforce without a parser.

}
// Key-free and already canonical: the engine publishes its own trust
// anchor. Leave it alone.
return ctrl.Result{}, nil

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two small asymmetries on the key-free canonical branch, both misconfig-only (no shipped engine declares <release>-tenant-ca as its source). It returns ctrl.Result{} with no RequeueAfter, so on the unwatched name-driven leg a self-source that later gains key material is never re-examined, whereas every other path here requeues on resyncInterval; and unlike the shared write path it does not assert the single-ca.crt/Opaque contract. It stamps no tenant-ca/tenantresource label, so it grants no tenant visibility on its own — but return ctrl.Result{RequeueAfter: resyncInterval} plus a warning on extra keys would make this corner symmetric with the rest.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both asymmetries are closed. The key-free canonical branch returns ctrl.Result{RequeueAfter: resyncInterval}, so a declaration-driven self-source that later gains key material is re-examined on the unwatched name leg rather than served with a key forever. And it no longer skips the contract: canonicalContractDeviation asserts the Opaque / single-ca.crt shape and warns on extra keys. Covered by TestReconcile_SourceAtCanonicalName_KeyFree_Resyncs and _ContractDeviation.

err := r.Reader.Get(ctx, types.NamespacedName{Namespace: ns, Name: name}, recorded)
switch {
case apierrors.IsNotFound(err):
// MERELY ABSENT. Hold the trust anchor and wait for the source to come

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness edge worth a look. If a declared source disappears and spec.caCert is then removed, resolveSource falls through to the empty label leg and returns nil, so pruneProjection runs — but because the recorded source is still absent it lands in this MERELY ABSENT branch and holds+requeues the projection indefinitely, even though removing the declaration is a definitive opt-out. The retired anchor stays tenant-readable until the source name reappears or the HelmRelease is deleted. The absent-vs-opt-out discriminator keys off the source Secret's existence, which cannot tell 'source rotating' apart from 'declaration withdrawn while the source happens to be gone'. Same-namespace and low-severity, but the withdrawal never fires.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed. pruneProjection now discriminates on the recorded source LEG, not the source's existence: a declared-leg projection (SourceModeAnnotation == modeDeclared) with no standing declaration is withdrawn without reading the source at all, so removing spec.caCert after the source has already vanished is a definitive opt-out and the retired anchor no longer lingers. Covered by TestReconcile_DeclarationRemovedAfterSourceVanished_WithdrawsProjection; the reissue case (source gone, declaration standing) still holds via TestReconcile_VanishedDeclaredSource_KeepsProjection.

# platform controller's own ServiceAccount; no tenant RBAC is widened.
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This widens the controller ServiceAccount from cluster-wide Secret read (already covered by the '*'/'*': get,list,watch catch-all) to cluster-wide create/update/patch/delete. It is genuinely required — projections land in tenant namespaces and RBAC cannot scope a verb to a name suffix — but it enlarges the blast radius if this SA is compromised, now co-located with cozystack.io/*: *. Non-blocking; worth a threat-model line, or later splitting the projection writes onto a dedicated SA that holds only the Secret verbs.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed it is inherent and non-blocking: RBAC cannot scope a verb to a name suffix and projections land in arbitrary tenant namespaces, so cluster-wide create/update/patch/delete on Secrets is required. I am treating it as an accepted residual; splitting the projection writes onto a dedicated ServiceAccount that holds only the Secret verbs is a reasonable later hardening, out of scope for this change.

failurePolicy is Fail: a Secret whose write cannot be evaluated is denied,
not admitted — fail-closed, as a trust-anchor control must be.
*/}}
{{- $certManagerWriter := "system:serviceaccount:cozy-cert-manager:cert-manager" }}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two non-blocking operational notes on this policy. The allowed writer is pinned as a literal, so a cert-manager repackage/rename must move it in lockstep; it fails closed (denies the new identity, killing the label leg) rather than open, and since postgres uses the declared leg the policy gates nothing live yet, so the drift would only surface when the first label-leg engine converges — deriving the identity from the cert-manager package values at render time would remove the seam. Separately, failurePolicy: Fail + [Deny] with no namespace/object selector means a Velero or kubectl restore of a labelled cert-manager Secret during DR is denied (the restorer is not cert-manager); cert-manager re-mints so it is recoverable, but it is worth a runbook note.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both are acknowledged as accepted operational residuals. The writer literal fails closed by design — a cert-manager rename denies the new identity rather than silently widening — and since the shipped engine is on the declared leg the policy gates nothing live yet, so the drift would only surface when the first label-leg engine converges. The DR case is recoverable because cert-manager re-mints the Secret. Deriving the identity from cert-manager package values at render time and a restore runbook note are fair follow-ups, out of scope here.

@lexfrei
Aleksei Sviridkin (lexfrei) force-pushed the feat/ca-extraction-controller branch from c5d95dc to 98df611 Compare July 15, 2026 10:32
@github-actions github-actions Bot added the area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review label Jul 15, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/controller/cacert/reconciler.go`:
- Around line 791-821: Update the projection reconciliation logic around
hasOwner to require exactly one OwnerReference matching owner, rather than
merely checking whether the desired owner exists. Treat any list with extra,
stale, or differing references as non-compliant and replace
existing.OwnerReferences with a single owner entry, while preserving the
early-return path only when labels, annotations, data, and ownership all match
exactly.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 4dc4b7c7-ff0d-4dbc-be35-e80204e60fe1

📥 Commits

Reviewing files that changed from the base of the PR and between c5d95dc and 98df611.

📒 Files selected for processing (15)
  • api/v1alpha1/applicationdefinitions_types.go
  • api/v1alpha1/zz_generated.deepcopy.go
  • cmd/cozystack-controller/main.go
  • internal/controller/cacert/postgres_wiring_test.go
  • internal/controller/cacert/reconciler.go
  • internal/controller/cacert/reconciler_test.go
  • packages/library/cozy-lib/templates/_tls.tpl
  • packages/system/application-definition-crd/definition/cozystack.io_applicationdefinitions.yaml
  • packages/system/cozystack-basics/templates/publish-ca-cert-writer-policy.yaml
  • packages/system/cozystack-basics/tests/publish-ca-cert-writer-policy_test.yaml
  • packages/system/cozystack-controller/templates/rbac.yaml
  • packages/system/cozystack-controller/tests/rbac_test.yaml
  • packages/system/postgres-rd/cozyrds/postgres.yaml
  • packages/tests/cozy-lib-tests/templates/tests/tls-cacert.yaml
  • packages/tests/cozy-lib-tests/tests/tls_cacert_test.yaml
🚧 Files skipped from review as they are similar to previous changes (9)
  • packages/system/application-definition-crd/definition/cozystack.io_applicationdefinitions.yaml
  • packages/tests/cozy-lib-tests/templates/tests/tls-cacert.yaml
  • internal/controller/cacert/postgres_wiring_test.go
  • packages/system/cozystack-basics/tests/publish-ca-cert-writer-policy_test.yaml
  • packages/library/cozy-lib/templates/_tls.tpl
  • api/v1alpha1/zz_generated.deepcopy.go
  • packages/system/cozystack-controller/tests/rbac_test.yaml
  • api/v1alpha1/applicationdefinitions_types.go
  • cmd/cozystack-controller/main.go

Comment thread internal/controller/cacert/reconciler.go

@IvanHunters IvanHunters left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The engine-agnostic cacert controller is fail-closed on every security-critical path: the declared source takes priority over the forgeable label leg, the write path copies exactly one whitelisted ca.crt and rejects any private-key/non-cert PEM, owner-ref GC is native, and Secret.type immutability is handled. The new VAP is a GA in-apiserver policy with no webhook pod (no cold-start trap) and rejects no existing Flux write (no chart writes the publish-ca-cert label). Additive CRD field with committed deepcopy/schema, no migration; cozyrds selects the key-free tenant-ca projection, so there is no TLS key leak.

Non-blocking follow-ups (already tracked): x509-parse the guard instead of header-match only; distinguish declaration-removed from source-temporarily-absent in pruneProjection; harden the cross-ns source lookup; the hardcoded cert-manager SA in the VAP is a documented fragile seam.

@myasnikovdaniil myasnikovdaniil left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — re-reviewed after the rebase onto current main. The core controller, _tls.tpl, the API types, the CRD, the postgres wiring, and the ValidatingAdmissionPolicy's rendered output are byte-identical to the previously-approved revision. The force-push added two targeted tests and two documentation clarifications, on top of the environment change from the wildcard-secret controller landing on main.

What I re-verified

  • The only behavioral change to this controller's environment is that the manager's shared Secret informer is now scoped (by the wildcard-secret controller) where it was previously unscoped. That makes the invariant "every Secret read goes through the uncached Reader or this controller's private, scoped caSourceCluster cache — never the shared cached Client" load-bearing rather than a review-only guarantee. Audited every Secret access: the five Secret reads (reconciler.go:613 via Cache; :677, :719, :869, :887 via Reader) all bypass the shared cache; the remaining cached Get/List calls (:424, :692, :1243, :1272) are HelmRelease / ApplicationDefinition, not Secrets; writes (:736, :830, :907) bypass the cache. The scoping does not regress this controller.
  • The two new tests (TestReconcile_NameDrivenSource_ReadThroughUncachedReader, TestReconcile_UpsertExistingCheck_ReadThroughUncachedReader) lock exactly that invariant; both pass. go test ./internal/controller/cacert/... and go vet are green, and helm unittest passes for the cozystack-basics VAP and the cozystack-controller RBAC.
  • The doc-only changes are accurate: the VAP comment explaining why DELETE is intentionally ungated (integrity-only policy; a DELETE-matching VAP would also gate the garbage-collector and namespace-controller and wedge namespace teardown), and the RBAC comment now attributing the shared Secret write grant to both reconcilers (verbs unchanged).
  • The two red CI checks are unrelated to this PR: Build packages/apps/kubernetes fails on an unresolvable upstream centos:stream9 image digest (that package is untouched here), and Analyze (go) was canceled mid-run after its Go build step passed.

Non-blocking follow-ups

  1. Single-owner enforcement on the projection (reconciler.go:795, :813) — see inline comment.
  2. Carried over from the earlier review, still applicable (these files are unchanged):
    • decode and x509.ParseCertificate the write-path guard instead of validating by PEM-header substring (reconciler.go:965, mirrored in _tls.tpl);
    • the declaration-withdrawn-while-source-absent prune corner leaves a retired anchor tenant-readable until the source reappears or the release is deleted (:890);
    • the key-free canonical branch returns no RequeueAfter and skips the single-ca.crt/Opaque assertion (:576);
    • selectorsDigest retains only the low 64 bits of the SHA-256 (:952);
    • applicationDefinition returns the first kind match without flagging duplicate kinds (:696);
    • the cluster-wide Secret write grant enlarges the controller ServiceAccount's blast radius (rbac.yaml);
    • the VAP's cert-manager writer is pinned as a literal, and a DR restore of a labelled source is denied (publish-ca-cert-writer-policy.yaml).

None of these block merge.

}
existing.Annotations[SourceRefAnnotation] = ref
existing.Annotations[SelectorsDigestAnnotation] = digest
if !hasOwner(existing.OwnerReferences, owner) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking hardening. The up-to-date short-circuit (:795) and this normalization both check only that the desired HelmRelease owner is present (hasOwner), not that it is the sole owner. A namespace actor able to update the projection could append a second, non-controller owner reference pointing at a long-lived object; it survives reconciliation, and once the HelmRelease is deleted the garbage collector keeps the Secret because that other owner still exists — the retired trust anchor lingers indefinitely.

Impact is bounded: the projection is key-free, same-namespace, and self-inflicted — the same class as the known limitation already noted in the PR description — so this is not a blocker. Cheap to close by gating both sites on len(existing.OwnerReferences) == 1 && hasOwner(...); worth a line in the known-limitations note either way.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both sites now gate on ownedSolelyBy (len(refs) == 1 && hasOwner), not bare hasOwner: a second appended owner reference fails that check, so the projection is re-homed back to exactly one controller ref and garbage collection cannot be kept alive by a squatted second owner. Covered by TestReconcile_ExtraOwnerReferenceIsStripped and TestOwnedSolelyBy.

IvanHunters
IvanHunters previously approved these changes Jul 16, 2026

@IvanHunters IvanHunters left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Reviewed with the cozy-review methodology (build + Go tests + helm unit tests for all three affected charts pass; load-bearing claims verified against sources).

No blocking defects. The change is additive: a new reconciler in the existing cozystack-controller binary creates a new Opaque <release>-tenant-ca Secret from the existing CNPG <release>-ca — nothing existing is mutated or pruned, so there is no Secret.type immutability trap and no resource-policy: keep needed. The secrets RBAC grant is pre-existing (WildcardSecretReconciler), not new. Fresh install adds no cold-start dependencies: the controller creates no Certificate, so no dependsOn on cert-manager is required, and the ValidatingAdmissionPolicy is inert on a cold cluster. make generate artifacts (CRD + deepcopy) are present.

Verified: the second cluster.Cluster is registered via mgr.Add and lands in the Caches runnable group gated by WaitForCacheSync (controller-runtime runnable_group.go), so the controller starts only after its cache syncs.

One non-blocking follow-up: the ValidatingAdmissionPolicy in publish-ca-cert-writer-policy.yaml hardcodes the allowed writer as system:serviceaccount:cozy-cert-manager:cert-manager, duplicating an identity whose source of truth is the cert-manager package (the author flagged this as a "FRAGILE SEAM"). Harmless today while the label-leg is dormant, but before the first label-leg engine converges it would be worth deriving the cert-manager namespace/SA from a shared value so a cert-manager repackaging can't silently break the seam in either direction.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@internal/controller/cacert/reconciler.go`:
- Around line 1578-1582: The hasOwner comparison omits BlockOwnerDeletion,
allowing blocking owner references to be treated as matches. Update hasOwner in
internal/controller/cacert/reconciler.go at lines 1578-1582 to compare
BlockOwnerDeletion alongside the existing OwnerReference fields, and add a test
case in internal/controller/cacert/reconciler_test.go at lines 1700-1723
verifying differing BlockOwnerDeletion values are not considered a match.

In `@packages/library/cozy-lib/templates/_tls.tpl`:
- Around line 185-198: Add an admission-policy gate for the
internal.cozystack.io/tenant-ca label so only the authorized publish-ca-cert
path may set it to true. Update the policy governing namespace Secret writers,
preserving the helper-rendered labels in the labels merge while preventing other
Secret writers from surfacing objects through the tenant read path.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 16628456-a3f3-4782-a7e5-569fa99278f3

📥 Commits

Reviewing files that changed from the base of the PR and between 98df611 and 377cd66.

📒 Files selected for processing (23)
  • .gitignore
  • api/v1alpha1/applicationdefinitions_types.go
  • api/v1alpha1/zz_generated.deepcopy.go
  • cmd/cozystack-controller/main.go
  • hack/check-publish-ca-cert-writer-pin.bats
  • hack/e2e-chainsaw/postgres/chainsaw-test.yaml
  • internal/controller/cacert/postgres_wiring_test.go
  • internal/controller/cacert/reconciler.go
  • internal/controller/cacert/reconciler_test.go
  • packages/apps/postgres/README.md
  • packages/apps/postgres/templates/dashboard-resourcemap.yaml
  • packages/apps/postgres/templates/db.yaml
  • packages/apps/postgres/tests/tenant_ca_rbac_test.yaml
  • packages/library/cozy-lib/templates/_tls.tpl
  • packages/system/application-definition-crd/definition/cozystack.io_applicationdefinitions.yaml
  • packages/system/cozystack-basics/templates/publish-ca-cert-writer-policy.yaml
  • packages/system/cozystack-basics/tests/publish-ca-cert-writer-policy_test.yaml
  • packages/system/cozystack-controller/templates/rbac.yaml
  • packages/system/cozystack-controller/tests/rbac_test.yaml
  • packages/system/postgres-rd/cozyrds/postgres.yaml
  • packages/tests/cozy-lib-tests/templates/tests/tls-cacert.yaml
  • packages/tests/cozy-lib-tests/tests/tls_cacert_test.yaml
  • packages/tests/cozy-lib-tests/tests/tls_cacert_values.yaml
🚧 Files skipped from review as they are similar to previous changes (9)
  • packages/tests/cozy-lib-tests/templates/tests/tls-cacert.yaml
  • packages/system/cozystack-controller/templates/rbac.yaml
  • packages/system/application-definition-crd/definition/cozystack.io_applicationdefinitions.yaml
  • packages/system/cozystack-basics/tests/publish-ca-cert-writer-policy_test.yaml
  • packages/system/cozystack-controller/tests/rbac_test.yaml
  • api/v1alpha1/applicationdefinitions_types.go
  • packages/system/postgres-rd/cozyrds/postgres.yaml
  • cmd/cozystack-controller/main.go
  • internal/controller/cacert/postgres_wiring_test.go

Comment thread internal/controller/cacert/reconciler.go
Comment thread packages/library/cozy-lib/templates/_tls.tpl
The only watches on the projection Secret were the source-Secret index (keyed on
spec.projections.sourceSecretName) and the sentinel itself. The projection is
named <release>.tenant-ca, which is no sentinel's source, so an in-place rewrite
of its ca.crt was corrected only by the five-minute resync — a window in which a
forged trust anchor is served.

Map the projection back to its owning sentinel through the controller owner
reference it carries, on the same dedicated metadata cache the source watch uses.
Owns(&corev1.Secret{}) would register against the manager cache, whose Secret
informer is label-scoped to the WildcardSecret replicas, so a projection carrying
no wildcard label would never be delivered — it would compile and pass a fake
client yet never fire in a real cluster.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
…ases

Two gaps in the sentinel-to-canonical-name lifecycle:

Withdrawal — the reconciler published each declared entry but never deleted
one. A sentinel that keeps existing with its CACert entry removed (a chart
gating it on tls.enabled, say) left the old <release>.tenant-ca served forever,
because owner-reference GC only fires when the whole sentinel is pruned. A
sentinel that reconciles with zero CACert entries now deletes the projection at
the canonical name, but only when it still carries that sentinel's owner
reference, so a foreign Secret or another sentinel's projection is never touched.

Contention — two sentinels in one namespace with the same release label both
resolve to the single canonical name. Whichever reconciled first would publish
an arbitrary CA and the two could flap ownership of one projection. Refuse both,
mirroring the more-than-one-CACert-entry refusal, so nothing is published until
the declaration lives on exactly one sentinel.

The ClusterRole comments claimed the reconciler never deletes a Secret; corrected
to describe the owner-scoped withdrawal delete.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
The projection carries two annotations whose keys still named a publish-ca-cert
discovery leg that no longer exists in the tree. Rename them to
internal.cozystack.io/ca-cert-source and internal.cozystack.io/ca-cert-selectors
while the API is unshipped; both stay functional (the selectors digest drives
the re-admission drift check, the source ref is a drift input and a traceability
record).

Resolving the ApplicationDefinition matched on Kind alone from a List of
unspecified order. Gate the match on the release's group label too, so a release
stamped with an unexpected group cannot resolve a same-kind definition by
accident.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
select-e2e.sh mapped reverse dependencies to Chainsaw suites only through
*-application sources and a hardcoded external-dns, so a change confined to the
tenant-projection writer policy (shipped by cozystack-basics) selected every
suite that depends on it but not cacert — whose Chainsaw is the policy's only
live deny/allow proof. Add the cozystack-basics reverse mapping, mirroring the
external-dns precedent, and pin both directions: a cozystack-basics change
selects cacert, and a cacert run installs cozystack-basics.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
The dependsOn comment warned that a capability gate renders the policy out at
first install and never re-adds it, which read as contradicting the
.Capabilities.APIVersions.Has guard the policy templates now carry. They do not
conflict: the warning is about gating on the matched types' CRDs, which may not
exist yet at first install, while the template guard names the ValidatingAdmission
Policy API itself — GA on every supported cluster, so always present. Spell out
that a maybe-absent matched-type CRD needs dependsOn ordering while an
always-present core API can be gated in the template.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
Both suites compared the projection's ca.crt against the source PEM byte for
byte. The controller re-encodes ca.crt from the DER it parsed, so the two match
today only because openssl and the re-encoder both emit header-free 64-column
PEM; a header line or column-width change upstream would redden a correct
projection. Compare the parsed certificates via openssl x509 -noout -fingerprint
instead, which is identity-stable across any re-encoding.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
The two-sentinel refusal returned without a RequeueAfter. NoRelease and
MultipleCACert can do that because their fix edits the same sentinel, which the
For() watch delivers; a contest is cleared by deleting the SIBLING sentinel, and
nothing enqueues this one when a sibling changes. So after an operator removed
the duplicate, the surviving sentinel withheld its anchor until the informer's
global resync — hours. Return RequeueAfter: resyncInterval on the contest branch
so recovery rides the resync backstop, and clarify that an anchor an earlier
uncontested reconcile already published is left in place rather than withdrawn.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
…draw note

The selectors digest kept only the first 8 bytes of the SHA-256. A collision
skips the re-admission the doc calls load-bearing — the revocation direction,
where a tenant would silently keep read access after a definition stopped
selecting the anchor — and the annotation has no length limit, so store the full
hash. Also correct the withdraw-branch comment: MinItems=1 prevents a
single-entry sentinel from emptying its list, so the branch is reached when a
future non-CACert entry replaces the CACert one, not by gating a lone entry off.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
withdrawProjection recognised its projection by an exact owner-reference
match while adoption uses a name match, so the two disagreed about what
counts as "ours" — and withdrawal was the stricter, which is backwards:
"safe to overwrite" is a stronger claim than "safe to delete".

Two projections a name match adopts and rewrites were read as foreign
by the exact match and left in place: one carrying a previous
incarnation's UID (delete-and-recreate under the same name), which
owner-reference garbage collection never reaps once the CACert entry is
gone, so the retired anchor stayed tenant-readable forever; and one
whose BlockOwnerDeletion flag had drifted, the very drift the adoption
path normalizes. Decide ownership with isOurProjection so both paths
answer the question the same way.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
An entry with an empty sourceSecretName sent the reconciler to Get a
Secret with an empty name. The apiserver answers that with "resource
name may not be empty" — not NotFound — so it escaped the not-found
branch, was returned from Reconcile, and wedged the sentinel on
exponential backoff. The error path returns before any status write, so
no Ready condition was ever recorded: the silent retry the sentinel was
introduced to make visible.

Pin sourceSecretName to MinLength=1 so a live apiserver rejects it at
admission, and guard the reconcile so an object admitted by an older CRD
resolves to Ready=False/SourceInvalid instead of an error.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
The package doc claimed there is no withdrawal logic and that a retired
declaration is only ever a deleted sentinel. withdrawProjection has
since become the second mechanism: it deletes a projection when the
sentinel outlives its CACert entry, which owner-reference garbage
collection never reaps because the owner still exists. State both paths.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
The ApplicationDefinition mapping fans out to every TenantProjection in
the cluster on every definition event, including Flux's periodic no-op
re-apply of the many *-rd definitions. The selectors-digest drift check
already suppresses the resulting writes, but not the load: each enqueued
reconcile still issues several uncached reads.

Gate the watch with a predicate that delivers an update only when
spec.secrets — the field the selectors digest is computed from — changed,
while letting creates and deletes through. Comparing the same field the
digest digests keeps the gate and the digest from drifting apart.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
A key-free Secret already sitting at the canonical trust-anchor name is
left untouched and reported Ready. But tenant visibility comes from the
internal.cozystack.io/tenant-ca label, and an engine-owned Secret at that
name carries none, so the lineage webhook marks it tenantresource=false
and the tenant is locked out while the controller reports "published".

Add the missing label to the canonical-name contract deviations so the
warning names it, and collect all deviations rather than only the first
so a single warning describes every way the object falls short.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
Go 1.26's extended new() builtin takes an expression, so new(true) and
new(false) replace ptr.To for the owner reference's Controller and
BlockOwnerDeletion flags. ptr.Deref, which has no builtin equivalent,
stays.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
@lexfrei
Aleksei Sviridkin (lexfrei) force-pushed the feat/ca-extraction-controller branch from 703b67c to 156ded1 Compare July 21, 2026 15:04
@lexfrei

Copy link
Copy Markdown
Contributor Author

All six addressed on the rebase. Withdrawal now uses the same name-match gate as adoption (isOurProjection), so the two stop disagreeing: the stale-UID and BlockOwnerDeletion-drift projections get withdrawn now, both pinned with tests. Empty sourceSecretName gets MinLength=1 plus a guard for objects an older CRD admits. I confirmed that one live on a 1.36 cluster: creating a sentinel with an empty sourceSecretName is rejected at admission now. The package doc, the ApplicationDefinition watch predicate on spec.secrets, the missing tenant-ca label deviation, and ptr.To to new() are all in.

Rebased onto main. Two later commits (kubevirt-csi base image) are orthogonal and touch none of these files, so I left them for the merge rebase.

New head 156ded1.

@lllamnyp

Copy link
Copy Markdown
Member

This is not another round of findings. The six from the last round are all fixed, each with the test that pins it, and two of them you found in your own work rather than in the review. I have nothing further on correctness.

What I want to raise instead is scope, and I want to be upfront that acting on it costs you a rebase and changes no behaviour whatsoever. It is worth doing anyway, and the reason is that this branch has grown by review: each round added a guard, a test and a comment, and none of the rounds removed anything, so the surface has been ratcheting up even while the sentinel rework was deleting two thousand lines from the middle of it. The clearest symptom is that the current round asks to delete hack/check-tenant-projection-writer-pin.bats, which an earlier round effectively asked for. That is the ratchet visibly reversing, and it is the point to stop adding and start cutting.

The measurement

Against the merge base the branch is 43 files, +5133/-74. internal/controller/cacert/reconciler.go is 1489 lines, of which 730 are comment-only and 85 are blank — 674 lines of code. The CRD types are 115 lines. So the controller-and-a-CRD at the centre of this is roughly 800 lines of code, which is the right size for what it does. Everything that makes the branch feel large is around it, not in it.

Three of the touched files have no functional change at all — packages/apps/postgres/templates/dashboard-resourcemap.yaml, packages/apps/postgres/templates/db.yaml and packages/core/platform/sources/cozystack-basics.yaml are comment-only diffs. packages/system/cozystack-controller/templates/rbac.yaml adds 67 lines to introduce four:

- resources: ["tenantprojections"]
  verbs: ["get", "list", "watch"]
- resources: ["tenantprojections/status"]
  verbs: ["get", "update", "patch"]

The remaining 63 are an essay about the blast radius of a cluster-wide Secret write grant that already existed for the WildcardSecret reconciler and is unchanged by this branch.

Proposed split

PR A — the controller and its CRD. Everything below keeps its current diff unless the note says otherwise.

Path Note
api/internalapi/v1alpha1/** as-is
packages/system/cozystack-controller/definitions/internal.cozystack.io_tenantprojections.yaml as-is (generated)
internal/controller/cacert/reconciler.go trim comments, see below
internal/controller/cacert/reconciler_test.go as-is
cmd/cozystack-controller/{main.go,main_test.go} as-is
packages/system/cozystack-controller/templates/rbac.yaml keep the four rule lines, cut the essay
packages/system/cozystack-controller/tests/rbac_test.yaml trim the rationale prose
packages/apps/postgres/templates/tenant-projection.yaml drop the comment block
packages/apps/postgres/tests/tenant_projection_test.yaml as-is
packages/system/postgres-rd/cozyrds/postgres.yaml as-is
packages/apps/postgres/README.md as-is, it is the user-facing contract
hack/e2e-chainsaw/postgres/chainsaw-test.yaml as-is, this is the real end-to-end proof
hack/update-codegen.sh, hack/e2e-chainsaw/README.md as-is

Reverted to the merge base and not carried anywhere:

  • packages/apps/postgres/templates/dashboard-resourcemap.yaml — comment-only. Separately, whether a per-app Role granting Secrets by name is still the right mechanism at all is a real question, given this branch's own argument that the label path through tenantsecrets is the better one — but that is its own issue, not this PR's.
  • packages/apps/postgres/tests/tenant_ca_rbac_test.yaml — 84 lines asserting that a resourceName is absent from a Role.
  • packages/apps/postgres/templates/db.yaml — comment-only.
  • hack/check-tenant-projection-writer-pin.bats — already under a delete recommendation.

PR B — cozy-lib trust-anchor helper. packages/library/cozy-lib/templates/_tls.tpl and packages/tests/cozy-lib-tests/**, about 407 lines. This is the render-time producer, a different mechanism from the controller, and it has no production callers — the name convergence is a one-line change and the rest is guard hardening that stands on its own.

PR C — VAP capability gate. The .Capabilities.APIVersions.Has wrapper on route-, gateway- and ingress-hostname-policy.yaml, their tests, vap-api-capability-gate_test.yaml, and the packages/core/platform/sources/cozystack-basics.yaml comment, about 80 lines. Three of those four policies have nothing to do with CA extraction; this is an install-robustness fix that happened to be noticed here.

PR D — the sentinel writer policy, about 209 lines plus the two policy steps in hack/e2e-chainsaw/cacert/chainsaw-test.yaml — and worth deciding whether to have at all, see below.

PR E — the dot-free release.prefix pattern. api/v1alpha1/applicationdefinitions_types.go and the regenerated cozystack.io_applicationdefinitions.yaml, 13 lines. Small, but it is a tightening of validation on an existing, cluster-scoped, user-extensible API: an out-of-tree definition with an underscore in its prefix starts failing admission on its next apply. Every in-tree prefix passes, so nothing ships broken, but it deserves its own ! and its own release-note line rather than riding along in a feature PR.

PR F — the .gitignore chore, 20 lines, already its own commit.

On the writer policy

Worth a deliberate decision rather than inheritance. Its own comment makes the case against itself: base tenant roles grant no verb on internal.cozystack.io at all — that is the group's whole reason for existing — so RBAC, not the policy, is the boundary. By that reasoning every CRD in the group would want one, and the platform has many resources tenants are not meant to write without a VAP each.

Dropping it also simplifies two things rather than relocating them. The cacert chainsaw suite currently has to impersonate flux to create a sentinel at all; without the policy that step becomes a plain apply. And the pinned-writer bats guard stops being a problem to solve, since the identity it pins is generated by flux-operator at runtime and cannot be cross-checked against any chart this repo can render.

If you want the defence in depth, it is a clean standalone PR. I would not block on either answer.

Mechanics

Do not rebase the existing commits — they interleave fixes across areas and replaying them by hand is where mistakes happen. Take the final tree and keep only the in-scope paths:

git checkout -b feat/ca-extraction-controller-v2 741a5acc4
git checkout <current-head> -- \
  api/internalapi internal/controller/cacert cmd/cozystack-controller \
  packages/system/cozystack-controller \
  packages/apps/postgres/templates/tenant-projection.yaml \
  packages/apps/postgres/tests/tenant_projection_test.yaml \
  packages/apps/postgres/README.md \
  packages/system/postgres-rd/cozyrds/postgres.yaml \
  hack/e2e-chainsaw/postgres hack/update-codegen.sh

That subset is 16 files and 3876 insertions as it stands today. The other PRs come off the same base by cherry-picking only their own paths; none of them depends on another, so they can land in any order.

If the cacert chainsaw suite loses its two policy steps, hack/select-e2e.sh and hack/select-install.sh only need their cacert mapping if the remaining controller step stays in that suite — otherwise those four lines and their bats coverage go with PR D too.

Comment trimming

The rule I would apply is whether the comment explains something the next person cannot derive from the code in front of them.

Keep: function contracts; why the write path rebuilds the payload from parsed DER instead of copying validated input; why the Secret watch needs its own cache rather than Owns() (the per-GVK cache routing is genuinely a trap and nobody will rediscover it); the marker-drop re-admission mechanism; and the dot in the canonical name.

Move to the design proposal: the naming history, the rejected alternatives, the records of what previous iterations got wrong, and every paragraph arguing against a design that is no longer in the tree. The 132-line package header is the single biggest win and almost all of it is archaeology.

The naming invariant in particular is smaller than its current footprint suggests. It plays no part in tenant selection — delivery is entirely by the internal.cozystack.io/tenant-ca label — and it is not a security property. It exists so the projection cannot land on a name an operator will generate, since every engine-generated name has the shape <prefix><app><suffix> and app names are DNS-1035 labels, which cannot contain a dot, while Secret names are DNS-1123 subdomains, which can. That is worth stating once, at the constant:

// projectionSuffix completes the canonical name "<release>.tenant-ca".
//
// The DOT is load-bearing. Engine CA Secrets are named <prefix><app><suffix>,
// and app names are validated as DNS-1035 labels, which cannot contain a dot
// (pkg/apis/apps/validation); Secret names are DNS-1123 subdomains, which can.
// So no engine can generate this name for any application, and the projection
// cannot collide with an operator-owned object — in either direction, and both
// are unrecoverable. Do not "tidy" the dot into a dash.
projectionSuffix = ".tenant-ca"

The same argument is currently restated in _tls.tpl, dashboard-resourcemap.yaml and tenant_ca_rbac_test.yaml; those copies leave with those files.

What is actually blocking

Only PR A needs to land for the feature to work, and it needs no further correctness work from me — the split is a repackaging of code that is already right. B through F are yours to land whenever suits you, and I will not hold the feature on any of them.

@lexfrei

Copy link
Copy Markdown
Contributor Author

Split done per the review. The controller and its CRD are #3407, the only piece the feature needs; the rest land independently:

Comment trimming applied to #3407, with the package-header archaeology moved to the design proposal. I will close this once #3407 merges.

Aleksei Sviridkin (lexfrei) added a commit that referenced this pull request Jul 23, 2026
…nant Secret (#3407)

## What this PR does

Adds an engine-agnostic controller that publishes a key-free
`<release>.tenant-ca` Secret (only `ca.crt`) for a managed application,
so a tenant can verify the application's TLS without ever seeing a
private key. The source is declared, not guessed: a chart renders a
namespaced `TenantProjection` sentinel (group `internal.cozystack.io`)
naming the CA Secret to lift; the controller watches the sentinel,
extracts `ca.crt`, and writes the projection owner-referenced to the
sentinel, so garbage collection is native. The projection carries the
`internal.cozystack.io/tenant-ca` label the lineage webhook turns into
tenant visibility. Postgres is the first consumer.

The trust boundary is RBAC: no tenant role grants any verb on
`internal.cozystack.io`, so a tenant cannot forge a sentinel. A
companion `ValidatingAdmissionPolicy` that pins the writer to
helm-controller ships separately as defence in depth.

This is the controller-and-CRD slice, split out from #3299 for focused
review. It is the only piece required for the feature to work; the
render-time helper, the writer policy, the capability-gate, and the
release-prefix validation land as their own PRs.

### Downstream repositories

Walked the trigger map against the diff. This adds an internal CRD and
controller and a sentinel to the Postgres chart; no app is added or
renamed and no `values.schema.json` changes, so the typed provider and
Ansible are unaffected. The one to confirm is the Postgres README edit —
if the website mirrors it, it needs a refresh.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:

### Release note

```release-note
feat(cozystack-controller): publish a key-free per-application CA trust anchor (`<release>.tenant-ca`) that tenants read to verify managed-application TLS, declared through a new `TenantProjection` CRD. Postgres is the first consumer.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added tenant-facing CA certificate projections for PostgreSQL
deployments.
* Tenant secrets now provide only the CA certificate, excluding private
keys.
  * Added support for configuring and monitoring tenant CA projections.

* **Documentation**
* Updated PostgreSQL TLS instructions to retrieve certificates through
tenant secrets.

* **Bug Fixes**
* Improved protection against invalid certificates, secret collisions,
and accidental exposure of private key material.
* Added validation to ensure projected certificates remain synchronized
with their sources.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Aleksei Sviridkin (lexfrei) added a commit that referenced this pull request Jul 23, 2026
)

## What this PR does

Constrains `ApplicationDefinition.spec.application.release.prefix` to
`^[a-z0-9-]*$`, which excludes the dot. Release names are
`<prefix><app>`, and the tenant CA trust anchor is projected to
`<release>.tenant-ca`; the dot is the separator that keeps that name
unreachable by any engine, so a prefix carrying a dot would break the
guarantee. Every in-tree prefix already conforms. Split out from #3299.

```release-note
feat(api)!: `ApplicationDefinition.spec.application.release.prefix` must match `^[a-z0-9-]*$` (dot-free). A definition with a prefix containing a dot or any other character is rejected at admission; correct it before applying this CRD.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Added validation for release prefixes to allow only lowercase letters,
numbers, and hyphens.
* Updated guidance explaining how prefixes affect generated release
names and tenant CA trust anchors.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
@lexfrei

Copy link
Copy Markdown
Contributor Author

Superseded by the split. The controller and CRD landed in #3407; #3409, #3411, #3412 merged alongside. #3408 and #3410 continue on their own review threads. Closing this in favour of the focused PRs.

Aleksei Sviridkin (lexfrei) added a commit that referenced this pull request Jul 23, 2026
…3409)

## What this PR does

Wraps the route, gateway, and ingress hostname
`ValidatingAdmissionPolicy` templates in a
`.Capabilities.APIVersions.Has` gate so they render only where the VAP
API is available. Without it, an operator-generated HelmRelease with
drift detection off renders the policies out at first install on a
cluster missing the API and never adds them back. Each template gains a
sibling test asserting it renders zero documents when the API is absent.
Install-robustness fix, split out from #3299.

```release-note
fix(cozystack-basics): hostname ValidatingAdmissionPolicies render only where the VAP API is available, so a first install on a cluster without it no longer drops them permanently.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Hostname validation policies now render only when the Kubernetes
admission policy API is available, preventing unsupported policy
resources on older clusters.
* **Tests**
* Added/updated unit tests to explicitly simulate both missing and
present admission policy API capabilities, ensuring templates render (or
not) as expected.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Aleksei Sviridkin (lexfrei) added a commit that referenced this pull request Jul 24, 2026
…ame (#3408)

## What this PR does

Ships `cozy-lib.tls.caCertSecret`, a render-time helper that emits a
key-free trust-anchor Secret carrying only `ca.crt`. It converges the
object name on `<release>.tenant-ca`, requires the whole value to be
certificate blocks so a stray private-key header or trailing bytes are
rejected, and coerces numeric scalars before the guard runs so a numeric
value cannot slip past.

No chart calls it yet. It is the render-time producer half of the
trust-anchor contract; the controller half is #3407, now merged. The
doc-comment on the helper shows the one-line call a chart uses, and says
outright there is no caller yet, so nobody reads it as wired when it is
not.

Split out from #3299. Comments trimmed on review.

```release-note
fix(cozy-lib): the CA trust-anchor helper emits the canonical `<release>.tenant-ca` name and rejects non-certificate input.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- Updated generated TLS CA Secret naming to use the
`<release>.tenant-ca` convention.
- Added/ensured a tenant CA label is set alongside the existing tenant
resource label.
  - Continued support for custom labels (caller labels still applied).

- **Bug Fixes**
- Strengthened CA certificate PEM validation to be fail-closed: requires
non-empty, complete `BEGIN/END CERTIFICATE` blocks only.
- Rejects private-key material and other malformed or improperly
structured PEM input, including unexpected extra content.

- **Tests**
- Updated TLS CA certificate fixtures and assertions to cover additional
negative cases and refined error expectations.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/platform Issues or PRs related to platform infrastructure (bundle, flux, talos, installer) area/uncategorized PR auto-labeler could not map title scope to a known area/*; please review kind/feature Categorizes issue or PR as related to a new feature size/XXL This PR changes 1000+ lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Engine-agnostic CA extraction controller for the ca.crt-only trust anchor

4 participants