Create a plan based on this documentation

[rocrisp]

Operator Testing Workflow: requirements for interfacing owner and Certification Tester

Steps After Installing an Operator to Deploy All Operands

1. Verify the operator CSV is Succeeded : confirm OLM installed it correctly
2. Check that all operator pods are running : controller-manager, webhooks, etc.
3. Apply the operator's Custom Resources (CRs) : triggers operand deployment
4. Wait for operand readiness : pods running, services available, status conditions healthy
5. Validate the deployed state : operands are functional, not just running
6. Run the test suite against the live operands

---

Why This Is Hard: Raw Challenges at Scale

Before defining the workflow, it's worth naming the real-world obstacles that make operator certification testing fundamentally difficult : especially when a single tester is responsible for 50+ operators across releases.

1. Special Hardware Requirements

Many operators require hardware that isn't available in generic CI clusters: PTP-capable NICs, SR-IOV devices, GPUs, FPGAs, bare-metal with specific BIOS settings, or even external equipment like a GNSS antenna or PTP grandmaster clock.

Mitigation: The operator owner should provide two CR profiles in their bundle:

7. full overlay : requires real hardware, exercises full functionality
8. noop / minimal overlay : deploys the operand with a no-op or simulated configuration so the tester can at least verify that the operator installs, creates its operands, and reaches a stable state : without needing the hardware

This lets the certification tester run structural and lifecycle tests on any generic cluster, and reserve hardware-dependent tests for dedicated lab runs.

2. Secrets, Licenses, and Special Keys

Some operators require credentials to function after installation:

9. Pull secrets for private operand images
10. License keys or entitlement certificates
11. API tokens for external services (e.g., cloud provider credentials, monitoring backends)
12. TLS certificates or CA bundles

These cannot be committed to a public repo and are awkward to share.

Mitigation: The bundle's metadata.yaml should declare required secrets by name, type, and purpose : but never contain the values. The tester's CI injects them from a vault:

requirements:
  secrets:
    - name: cloud-api-token
      namespace: openshift-operator-ns
      type: Opaque
      keys: ["API_KEY"]
      description: "Token for the external monitoring API"
      how_to_obtain: "Request from operator-owner@example.com or see vault path secret/ops/cloud-api-token"

3. Tracking Changes Across Releases

Operators evolve every release: CRD schemas change, new operands are added, default values shift, and deprecated fields are removed. A CR that worked in 4.15 may silently break in 4.16.

At 50+ operators, no tester can manually track these diffs.

Mitigation:

13. The test bundle is versioned in the operator's repo and tagged per release. A branch or tag like test-bundle/v4.16 makes changes trackable via git diff.
14. metadata.yaml includes a bundle_version and min_ocp_version field so the CI runner can detect mismatches automatically.
15. Owners should run their own bundle in their own CI as a gate : if they break their own bundle, they find out before the certification tester does.

operator:
  name: ptp-operator
  bundle_version: "2.1.0"
  min_ocp_version: "4.16"
  max_ocp_version: "4.17"

4. Debugging Failures Without Domain Expertise

When an operand fails to deploy, the tester sees generic symptoms: a pod in CrashLoopBackOff, a CR stuck in Progressing, or a webhook rejection with a cryptic message. Diagnosing these requires deep operator knowledge.

Even with AI-assisted debugging, the tester cannot be expected to reason about 50+ operators' internal state machines, dependency chains, and failure modes.

Mitigation:

16. The 02-validate.sh script should produce structured, actionable error messages : not just "failed," but "PtpConfig did not reach Ready because no node with label X was found."
17. The bundle should include a troubleshooting.md or a known_issues section in metadata.yaml:

known_issues:
  - symptom: "linuxptp-daemon pod in CrashLoopBackOff"
    cause: "No PTP-capable NIC detected on scheduled node"
    fix: "Ensure nodes are labeled with feature.node.kubernetes.io/ptp"
  - symptom: "CSV stuck in InstallReady"
    cause: "Missing OperatorGroup in target namespace"
    fix: "Apply 00-prerequisites.yaml before creating the Subscription"

18. A must-gather command should be declared so the tester can collect diagnostics and hand them to the owner without back-and-forth:

diagnostics:
  must_gather_image: "quay.io/openshift/ptp-must-gather:v4.16"

5. The Scale Problem: 50+ Operators × N Releases

This is the compounding factor. Every challenge above is manageable for one operator. At 50+, it becomes untenable without standardization:

Challenge	At 1 operator	At 50+ operators
Learning CR schemas	Feasible	Impossible
Tracking release changes	Manual diff	Must be automated
Debugging failures	Slack the owner	Needs self-service diagnostics
Procuring special HW	One lab booking	Combinatorial lab scheduling
Managing secrets	One vault entry	Requires a secrets inventory

This is why the standardized bundle format is non-negotiable. Without it, every operator is a bespoke snowflake, and the certification tester becomes a bottleneck who spends more time on Slack than on testing.

Summary of Mitigations

Challenge	Owner Provides	Tester Gets
Special hardware	noop/minimal CR overlay	Can test lifecycle on any cluster
Secrets & keys	Declared in metadata.yaml with instructions	Injects from vault, no guessing
Release drift	Versioned + tagged bundle, bundle_version field	CI detects mismatches automatically
Debugging without expertise	Structured errors, known_issues, must-gather image	Self-service diagnostics, no Slack
Scale (50+ operators)	All of the above, in a standard format	One generic CI pipeline for all operators

---

Separation of Responsibilities

Responsibility	Operator Owner	Certification Tester
Knows	CRD schema, valid CR configs, HW requirements, health signals	Test framework, cluster provisioning, test execution, reporting
Provides	A "test bundle" of reference CRs + validation logic	Cluster with required HW profile, OLM install, test harness
Does NOT need to know	How the test infra works, Prow config, CI plumbing	CRD internals, tuning parameters, operand architecture

---

The Interface Contract: A Test Configuration Bundle

The operator owner provides a single, self-contained artifact that the certification tester consumes without needing operator expertise.

Bundle Structure

operator-test-bundle/
├── 00-prerequisites.yaml      # Namespaces, RBAC, ConfigMaps, Secrets
├── 01-operand-crs.yaml        # The Custom Resources to apply (the core handoff)
├── 02-validate.sh             # Script that exits 0 when operands are healthy
├── metadata.yaml              # Machine-readable metadata (see below)
└── teardown.yaml              # Cleanup resources (optional)

metadata.yaml : The Machine-Readable Contract

operator:
  name: ptp-operator
  channel: stable
  namespace: openshift-ptp
  catalogSource: redhat-operators

requirements:
  hardware:
    - label: "feature.node.kubernetes.io/ptp"
      description: "Node with PTP-capable NIC (e.g., Intel E810)"
      min_nodes: 1
    - label: "node-role.kubernetes.io/worker"
      min_nodes: 2
  operators:
    - name: nfd-operator
      namespace: openshift-nfd

health_check:
  timeout_seconds: 300
  conditions:
    - apiVersion: ptp.openshift.io/v1
      kind: PtpConfig
      name: grandmaster
      namespace: openshift-ptp
      jsonpath: ".status.conditions[?(@.type=='Ready')].status"
      expected: "True"

---

Automating the Handoff

A generic test runner consumes the bundle without operator-specific knowledge:

Automated Flow

1. Tester's CI provisions cluster (using HW labels from metadata.yaml)
2. Tester's CI installs prerequisite operators (from metadata.yaml)
3. Tester's CI installs the target operator (from metadata.yaml)
4. Tester's CI applies:  kubectl apply -f 00-prerequisites.yaml
5. Tester's CI applies:  kubectl apply -f 01-operand-crs.yaml
6. Tester's CI runs:     ./02-validate.sh  (or evaluates health_check from metadata)
7. Tester's CI runs:     certification test suite
8. Tester's CI applies:  kubectl delete -f teardown.yaml

The tester never needs to understand the CRs : they just apply them. The owner never needs to understand the CI pipeline : they just provide the bundle.

---

What Makes This Work in Practice

1. The bundle lives in the operator's repo

Versioned alongside the code so it's always in sync. Suggested path: test/certification/ or config/test-bundle/.

2. The validate script is the key

Without it, the tester has no way to know when operands are truly ready. Only the owner knows what "healthy" means for their operator.

3. metadata.yaml replaces tribal knowledge

Instead of Slack messages saying "you need an E810 NIC," the HW requirements are machine-readable and can drive cluster provisioning automatically.

4. Parameterization for different environments

Use kustomize overlays or simple envsubst for environment variance:

operator-test-bundle/
├── base/
│   ├── 01-operand-crs.yaml
│   └── kustomization.yaml
├── overlays/
│   ├── baremetal/
│   │   └── kustomization.yaml
│   └── virtual/
│       └── kustomization.yaml

The same bundle works across lab configurations without the tester hand-editing CRs.

---

Summary

What	Who Provides	Format
Reference CRs to deploy operands	Operator owner	YAML files in a test bundle
HW/SW prerequisites	Operator owner	metadata.yaml (machine-readable)
Health check / readiness validation	Operator owner	Script or declarative conditions
Cluster provisioning + OLM install	Certification tester	CI pipeline reading metadata.yaml
Test execution + reporting	Certification tester	Test harness consuming the bundle

The single most impactful thing to standardize is the metadata.yaml schema : once that contract exists, everything else can be automated. The operator owner fills it in once, and every tester in the organization can consume it without a single Slack conversation.