Introduce the reflector to reduce the k8s API calls

[yairpod]

As requested in issue 251.
This PR replaces the direct k8s API calls with reflectors.
This should reduce the API server network latency and consumption of resources on the API server and etcd as well as risks hitting rate limits (HTTP 429) during heavy workloads.
The cost is increased memory usage as we cache the information from the API server (the CPU for the watcher threads should be reasonably low).
I have not been able to test the cache memory usage as a kind cluster will have a tiny memory signature anyway.

To test the API call reduction I have run the Integration tests on a clean kind cluster and run the command kubectl get --raw /metrics | grep apiserver_request_total | grep trusted, then summed the all calls of all types.
The unmodified operator had a total of 788 calls.
The PR code had a total of 463 calls.

Summary by Sourcery

Introduce shared reflector-backed caches for custom resources and core objects to reduce direct Kubernetes API usage and wire them into the operator controllers.

New Features:

• Add reflector-based in-memory stores for TrustedExecutionCluster, Machine, AttestationKey, Secret, and Deployment resources and wait for their initial synchronization before starting controllers.

Enhancements:

• Refactor attestation key controllers to share a common context carrying Kubernetes client and reflector stores instead of repeatedly listing resources from the API server.
• Use the reflector-backed TrustedExecutionCluster store in the main reconcile path to detect non-unique clusters without performing a fresh list call.
• Update trustee attestation key volume management to derive secrets and deployment state from reflector caches rather than live API reads.
• Adjust and extend tests to work with pre-populated reflector stores and reduced API interactions, while keeping existing behavior coverage.

[sourcery-ai]

Reviewer's Guide

Replaces direct Kubernetes API list/get calls with cached reflectors and shared in-memory stores for TEC, Machine, AttestationKey, Secret, and Deployment resources, wiring these stores into controllers and tests to reduce API traffic while ensuring caches are synced before starting controllers.

File-Level Changes

Change	Details	Files
Introduce reflector-based caches for key resources and wire them into the main controller context.	Add reflector Store to ClusterContext and use it in reconcile to enforce single-cluster constraint without live List calls. Instantiate reflector Store+Writer pairs for TrustedExecutionCluster, Machine, AttestationKey, Secret, and Deployment resources in main. Spawn background reflector tasks backed by watcher streams for each resource type and wait for initial cache sync with a timeout before starting controllers.	operator/src/main.rs
Refactor attestation-key controllers to use shared AkContextData with reflector-backed stores instead of listing from the API server.	Introduce AkContextData struct encapsulating Client and Stores for Machine, AttestationKey, Secret, and Deployment. Change ak_reconcile and machine_reconcile to iterate over cached store state instead of issuing List calls. Extend approve_ak to accept a Secret store and use it to check for existing secrets instead of a live Get call. Change secret_reconcile and controller launch functions to accept and propagate AkContextData rather than bare Client.	operator/src/attestation_key_register.rs
Make trustee deployment/secret volume update logic consume reflector caches instead of live API requests.	Change update_attestation_keys to derive attestation-key Secret names from the Secret Store state rather than listing via the API. Change update_attestation_keys to fetch the trustee Deployment from the Deployment Store cache by ObjectRef instead of a live Get call. Update call sites to pass in Secret and Deployment Stores when updating trustee deployment volumes.	operator/src/trustee.rs
Update tests to work with reflector-based stores instead of direct API list expectations.	Create helper two_cluster_tec_store to build a pre-populated Store with two TrustedExecutionClusters for tests. Adjust reconcile_non_unique and reconcile_error tests to inject the pre-populated Store and pre-set uid so they assert only on the PATCH call, reducing expected API interactions. Initialize a dummy Store in dummy_cluster_ctx used by other tests.	operator/src/main.rs

Possibly linked issues

• Introduce the reflector to reduce the k8s API calls #251: PR introduces reflectors and cached stores for TEC, Machine, AK, Secret, and Deployment, replacing direct get/list calls.

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[sourcery-ai]

Hey - I've found 1 issue, and left some high level feedback:

• The reflector tasks currently discard all events with .for_each(|_| async {}); consider at least logging or handling errors from the reflector stream so that broken watches or deserialization issues are visible rather than silently ignored.
• In update_attestation_keys, returning an error when the trustee Deployment is not found in the cache (ok_or_else) is a behavioral change from the previous live-GET; if the deployment may legitimately not exist yet or be temporarily absent from the cache, consider treating this as a no-op with a log message instead of a hard error.

Prompt for AI Agents

Please address the comments from this code review:

## Overall Comments
- The reflector tasks currently discard all events with `.for_each(|_| async {})`; consider at least logging or handling errors from the `reflector` stream so that broken watches or deserialization issues are visible rather than silently ignored.
- In `update_attestation_keys`, returning an error when the trustee `Deployment` is not found in the cache (`ok_or_else`) is a behavioral change from the previous live-GET; if the deployment may legitimately not exist yet or be temporarily absent from the cache, consider treating this as a no-op with a log message instead of a hard error.

## Individual Comments

### Comment 1
<location path="operator/src/attestation_key_register.rs" line_range="298-299" />
<code_context>
     info!("Secret reconciliation for AttestationKey secret: {secret_name}");

-    let secrets: Api<Secret> = Api::default_namespaced(Arc::unwrap_or_clone(client.clone()));
+    let secrets: Api<Secret> = Api::default_namespaced(ctx.client.clone());
     finalizer(&secrets, ATTESTATION_KEY_SECRET_FINALIZER, secret, |ev| async move {
         match ev {
             Event::Apply(_secret) => {
</code_context>
<issue_to_address>
**issue (bug_risk):** The `ctx` Arc is captured by reference into the finalizer closure, which is likely required to be `'static` and can lead to lifetime issues.

Here `finalizer` likely requires a `'static` closure (e.g., it spawns or stores it). Because the closure is declared with `|ev| async move { ... }`, `ctx` is currently captured by reference from the outer function, which will not satisfy `'static`.

Instead, clone the `Arc` and move it into the closure so it is owned by the closure environment, e.g.:

```rust
let ctx = ctx.clone();
finalizer(&secrets, ATTESTATION_KEY_SECRET_FINALIZER, secret, move |ev| {
    let ctx = ctx.clone();
    async move {
        match ev {
            Event::Apply(_secret) => {
                trustee::update_attestation_keys(
                    ctx.client.clone(),
                    &ctx.secret_store,
                    &ctx.deployment_store,
                )
            }
            // ...
        }
    }
})
```

This way the closure owns an `Arc<AkContextData>` and can meet the `'static` bound.
</issue_to_address>

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.}

[Jakob-Naucke]

• don't worry about the CI failure, quay.io is having trouble. That said, ensure your code has been run through formatting (e.g. cargo fmt)
• is this also possible for ApprovedImages (reference_values.rs:331)?
• commit message nits: use infinitive (Replace direct kube…) remove Subject:

[Jakob-Naucke]

still wondering if it can also be implemented for ApprovedImages (reference_values.rs:331)?

[Jakob-Naucke]

I think by that point you could just pass a reference to the context

[alicefr]

@yairpod try to use Opus as model instead of Sonnet. It will give you much better result

[Jakob-Naucke]

Can the reflector also be used for reference values?

[Jakob-Naucke]

nit: use infinitive in the commit message subject (Replace …)

[Jakob-Naucke]

LGTM, would still like @alicefr's opinion

[alicefr]

if this can be caused by the CRD to not be installed we can error out earlier by getting the CRD. It will also verify that the API server is reachable or not. We can improve this in a follow up PR though.

[yairpod]

Sounds good for a follow up improvement

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: alicefr, Jakob-Naucke, yairpod

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment