docs: LLD for ValidationRun CRD (for #139)

[bdchatham]

Summary

Captures the council-driven low-level design for ValidationRun — a new namespaced CRD (validation.sei.io/v1alpha1) that orchestrates ephemeral-chain validation workloads on Harbor.

• One CRD, three reserved kinds — ValidationRun (implemented), ValidationSuite and ValidationSchedule (CRDs ship with v1, no reconcilers; reserved so future kinds don't paint v1 into a corner).
• Discriminator union at spec.type: LoadTest (v1) / SequenceTest / IntegrationTest (future). Avoids the Testkube TestWorkflow consolidation lesson.
• Chain abstraction at spec.chain.{validators,fullNodes} embeds SeiNodeDeploymentSpec directly (both required) — the controller materializes both fleets as owned children with chainId/role-discriminator/peer-selector injection. Cascade delete via OwnerReferences.
• Validation rules are alert + query against Prometheus, continuously polled per runProperties.interval (default 30s). One Prometheus client serves both types — alert rules use the synthetic ALERTS series. Observability-as-test-oracle: verdict = workload_exit_code AND ⋀rules.
• monitor-run plan task is the central refinement: a single async polling loop combining workload-monitoring + per-rule evaluation in each iteration. Conditions[TestComplete] is the workload-completion boundary; runProperties.stopOnFailure enables cooperative Job cancel on rule trip.
• Same-namespace by construction with one narrow exception (alert.ruleRef into label-allowlisted monitoring namespaces). Tenants pre-provision ServiceAccounts; the controller is never an IAM controller.
• Workload contract parity with sei-protocol/platform#235 — env vars, exit codes (0/1/2), ${RESULT_DIR} artifacts, .status.report.s3Url only (no termination-message echo into status).
• Plan integrates with existing internal/planner/ — same Build → Persist → Execute → Complete/Fail lifecycle, single-patch model, condition ownership on planner.

Council process

This LLD is the artifact of a /council Component-tier design pass. Process documented in the local checkpoint (not committed; lives at .council/workstream.yaml).

• Round 1 (parallel): product-manager (scope cuts), opentelemetry-expert (rule semantics + controller observability), platform-engineer (Harbor integration / RBAC / IRSA / GitOps)
• Round 2: kubernetes-specialist primary LLD author synthesizing Round 1 inputs
• /coral mid-council escalation: user asked late whether to abandon custom CRD for Argo Workflows. All four specialists weighed in independently; unanimous Path X (custom CRD). The load-bearing reason: Argo cannot natively own non-Pod CRDs cascade-style, and SND-as-cascading-child is the central organizing primitive of this design. Argo Workflows re-evaluation triggers documented in Future Work.
• Gate close 2026-04-28: 18 resolved one-way-door decisions documented in the LLD's "Resolved one-way-door decisions" section. Notable: API group validation.sei.io, discriminator spec.type, embed SeiNodeDeploymentSpec, hard-reject reserved env conflicts via CEL, drop mode field (continuous-only), drop .status.report.raw (S3 is authoritative), spec immutability via CEL, Tekton-style Succeeded + TestComplete conditions, Failed vs Error distinct phases.

Test plan

• Cross-review by kubernetes-specialist, platform-engineer, product-manager, opentelemetry-expert (council reviewers); resolve any MISMATCH/MISSING findings before merge
• LLD answers all six open questions from Design ValidationRun CRD for ephemeral-chain validation workloads on Harbor #139 — see "Six open questions from Design ValidationRun CRD for ephemeral-chain validation workloads on Harbor #139 — answers" table
• LLD enumerates and avoids all five OSS one-way-door warnings from Design ValidationRun CRD for ephemeral-chain validation workloads on Harbor #139 — see "Five one-way-door warnings from OSS survey — how design avoids each" section
• LLD's 18 resolved gate decisions match what was approved 2026-04-28 — verify "Resolved one-way-door decisions" section
• Design serves both Phase 1 consumers (seiload nightly + qa-testing on Harbor) without per-workload special cases
• Workload contract from sei-protocol/platform#235 referenced verbatim as the authoritative spec.loadTest.workload envelope
• Implementation handoff issues filed (separate workstreams; not part of this PR's acceptance) — types + planner tasks + reconciler + monitor-run polling + open-dependencies follow-ups
• Companion sub-issues filed for the seven open dependencies surfaced in the LLD (notably: SND readiness includes catching_up via sidecar /health probe; monitoring/ namespace gets sei.io/validation-shared-rules=true label; PodMonitor for sei-k8s-controller itself; heartbeat PrometheusRule)

Out of scope for this PR

• Implementation (the LLD does not authorize implementation; handoff issues drive that work)
• ValidationSuite and ValidationSchedule reconcilers (CRDs ship with v1; controllers are deferred)
• SequenceTest and IntegrationTest discriminator bodies (designed-as-extension-points, not implemented)
• Cross-component changes to SeiNodeDeployment readiness semantics (filed as a companion sub-issue against the SND/SeiNode reconciler maintainer)

References

• Source issue: Design ValidationRun CRD for ephemeral-chain validation workloads on Harbor #139
• Companion Phase 1 workstream: sei-protocol/platform#235
• Repo's prior LLD style template: docs/design/composable-genesis.md (matched here)

🤖 Generated with Claude Code

[bdchatham]

Revision pushed (commit ef3de08)

Material LLD revision following continued council session refinement after gate close. Spec shape generalizes; controller architecture introduces sub-controller rails for v2 actor expansion.

Schema changes

• chain.deployments[] (named map of SeiNodeDeploymentSpec) replaces chain.{validators, fullNodes}. CEL enforces "exactly one validator + ≥1 fullNode role." Child SNDs named {chainId}-{deploymentName}.
• Drop spec.type discriminator. Replace with composable optional blocks spec.load, spec.sequence (v2 reserved), spec.chaos (v2 reserved). Multiple actors can coexist (e.g., load + sequence concurrently). Rules-only Runs are valid.
• Drop endpointPolicy; fullNodes-fleet endpoints always used.
• IntegrationTest collapses into "load + rules with workload-as-verifier" (no separate kind needed).

Controller architecture

Two sub-controllers in the same binary, registered via Helm-opt-in:

• OrchestrationReconciler (always-on): chain bring-up, monitor-task-completion gateway, finalize
• LoadGenerationReconciler (Helm-opt-in, default-on): workload Job lifecycle

Coordination is conditions-as-IPC with field-manager isolation:

• Conditions[TestRunning] — gate (Orchestration writes)
• Conditions[LoadComplete] — actor signal (LoadGen writes)
• Conditions[TestCancelled] — cooperative halt (Orchestration writes)
• Conditions[Succeeded], Conditions[TestComplete] — terminal (Orchestration writes)

Single-writer table in the LLD documents which controller owns each condition + status field.

v2 expansion is purely additive — Sequence, Chaos, etc. each land as new sub-controllers, predicate-gated identically, no refactor of v1 code.

.status.plan (singleton) → .status.plans.{orchestration, loadGeneration} named map. Each controller writes only its own plan slot.

Architectural decisions baked in (Coral-validated)

• pods/exec is REJECTED for the v2 sequence plan. One-way door. Sequence steps submit chain transactions via short-lived Jobs against the RPC service.
• Chaos plan requires namespace-label gate sei.io/chaos-allowed=true (admission-checked) + --enable-chaos-plan compile-time flag.
• TaskPlan.TargetPhase decoupling for v1: ValidationRun controllers ignore the SeiNodePhase-typed field and own phase transitions in the finalize task. Avoids generalizing the shared type until a third controller needs it.
• monitor-task-completion (renamed from monitor-run) uses RequeueAfter for transient errors, never TerminalError. Status patches use optimistic concurrency, not blind SSA.
• .status.failedPlan reserved in v1; v2 reconcilers stamp their own slot for 2am on-call debugging.

Resolved one-way-door decisions table

Grew from 18 to 26 rows. New entries cover composable-blocks shift, deployments[] generalization, two-controller architecture, predicate-gated event delivery, field-manager isolation, TargetPhase decoupling, Helm opt-in, pods/exec rejection, chaos namespace gate, and .status.failedPlan reservation.

Tensions worth explicit reviewer attention

1. TaskPlan.TargetPhase decoupling is a v1 dodge, not a fix. v2 may force the shared-type generalization when SequenceReconciler ships. LLD documents this in Open Dependency fix: pass UseLocalSnapshot to configure-state-sync task #2.
2. TestComplete semantic shifted — in the prior draft it meant "Job reached terminal." It now means "Run finalized" (Orchestration-owned). External watchers expecting the prior semantic must switch to LoadComplete. Conditions tables document the new ownership.
3. Field-manager isolation is a runtime-discipline invariant, not a compile-time guarantee. Code review should verify each controller writes only its owned status fields.
4. Length grew to ~12,400 words (target was 9000-11000). Architectural section + 26-row decisions table drove the growth. Could compress on request.

Stat

1369 lines diff (819 insertions, 550 deletions). 1610 total lines, ~12,400 words.

[bdchatham]

Refinement pushed (commit df7e0f8)

Two small refinements per user direction:

1. Naming consistency. All validation reconciler types now carry the Validation prefix to distinguish them from the always-on SeiNode / SeiNodeDeployment defaults:

Old	New
OrchestrationReconciler	ValidationOrchestrationReconciler
LoadGenerationReconciler	ValidationLoadGenerationReconciler
SequenceReconciler (v2)	ValidationSequenceReconciler
ChaosReconciler (v2)	ValidationChaosReconciler
OrchestrationPlan	ValidationOrchestrationPlan
LoadGenerationPlan	ValidationLoadGenerationPlan

2. Opt-in deployment, mechanism-agnostic. The prior draft prescribed a Helm-chart-specific values.yaml shape. Since the chart isn't shipping in v1, that framing was premature. Refinements:

• Section retitled: Controller registration and opt-in deployment (was Helm chart and controller registration)
• Specific values.yaml YAML dropped; replaced with a mechanism-agnostic Go sketch
• Architectural property locked in: the validation slice is opt-in at deployment time; SeiNode / SeiNodeDeployment controllers are the always-on default; node operators running production validators don't carry the validation machinery
• Specific opt-in mechanism (values flags, build tags, env vars, separate Deployment manifests, or any combination) deferred to implementation
• One-way-door table row 23, future work entries, and open dependency feat: use explicit task status from sidecar for polling #8 all reframed accordingly

Stat: 248 lines diff (113 insertions, 135 deletions). 1588 total lines (down from 1610).

[bdchatham]

Companion sub-issues filed

The seven Open Dependencies surfaced in the LLD's "Open dependencies" section are now tracked as separate workstreams. Implementation of the ValidationRun controllers themselves (this PR) is independent of these — they're cross-cutting hygiene + platform-side work that benefits this PR plus other consumers.

sei-k8s-controller:

• SeiNode readiness probe should include catching_up=false on full-node-role nodes #144 — SeiNode readiness probe should include catching_up=false on full-node-role nodes
• Admission validation: reject SeiNodeDeployment with genesis on full-node-role deployments #145 — Admission validation: reject SeiNodeDeployment with genesis on full-node-role deployments
• Decide on TaskPlan.TargetPhase generalization vs per-controller phase ownership #146 — Decide on TaskPlan.TargetPhase generalization vs per-controller phase ownership
• Verify all controller status patches use optimistic concurrency, not blind SSA #147 — Verify all controller status patches use optimistic concurrency, not blind SSA

sei-protocol/platform:

• sei-protocol/platform#243 — harbor: add PodMonitor for sei-k8s-controller deployment
• sei-protocol/platform#244 — harbor: add heartbeat PrometheusRule for consecutive ValidationRun failures
• sei-protocol/platform#245 — harbor: label monitoring/ namespace with sei.io/validation-shared-rules=true

None of these block this PR's review. They land independently and converge with the controller implementation when it ships. Implementation handoff issues for the controllers (types + deepcopy, ValidationOrchestrationReconciler, ValidationLoadGenerationReconciler, etc.) will be filed once the LLD direction here is locked by review feedback.

[bdchatham]

Superseded — pivot to CLI substrate (sei-protocol/seictl#96)

Following implementation start, the value question surfaced: what does putting validation orchestration into a CRD actually buy?

Honest answer after re-evaluation: GitOps applicability + resilience to controller restart mid-run. Both are deliverable from a CLI substrate. Everything else this LLD's shape was paying for (declarative desired state, edge-triggered reconciliation, status partitioning, two cooperating reconcilers, phase machine, condition machine) is paying for semantics that test orchestration doesn't actually need — tests are imperative, time-bounded, and one-shot.

The replacement design is captured at sei-protocol/seictl#96 (`docs/design/validation-substrate.md`). Highlights:

• CLI primitives (`chain` / `rpc` / `load` / `harness` / `rules`) composed by sugar verbs (`bench`, `qa`, `shadow`) — same workload runtime contract from sei-protocol/platform#235, already shipped in `seictl bench up`.
• Single binary, two install paths: standalone `seictl` AND kubectl plugin via `kubectl-sei` symlink. MCP graduation works on the same parser.
• v1 ships effectively zero new code. Today's `bench up` covers the seiload-nightly use case (this LLD's primary Phase 1 consumer). Primitives land on demand with named triggers.
• Label-driven cascade-delete, not OwnerRefs across primitives. `sei.io/chain-id` selector across resource kinds.
• `rules watch` is a Job, not a controller — deferred until a real "passed-but-validators-OOM" signal.

What survives from this LLD

• The workload runtime contract (envs, exit codes 0/1/2, `${RESULT_DIR}`, S3 path convention) — kept verbatim from platform#235; already implemented.
• The `ValidationRule` schema (alert + query types) — copies into `seictl/cluster/internal/rules/` when the `rules watch` Job materializes.
• The chain spec shape (validators + fullNodes, embedded SND fields) — input to future `chain up` / `rpc up` verbs.
• `LabelPeerSource` mechanism — the new design rides this directly (no new SND fields).

What dies

• `ValidationRun` / `ValidationSuite` / `ValidationSchedule` CRDs (kinds reserved here are released)
• The two-reconciler dance, plan persistence in `.status`, condition machine, phase machine
• Discriminator-union `spec.type` — each composite IS its own verb; no `--type` flag needed
• The 18 resolved one-way-door decisions that were CRD-shape-specific (notably the spec immutability via CEL, Tekton-style condition vocabulary, `Failed` vs `Error` phase distinction) — most become moot at the CLI layer

Process notes for future work

The new design ran a tight coral round (platform-engineer / product-manager / product-engineer in parallel) rather than full council. The reasoning: this is one CLI's surface, not multi-component cross-cutting work. The LLD here was correctly council-tier because it spanned controller types + planner internals + observability contract; the CLI replacement is component-tier and lighter in surface.

The merged 1588 lines of design here aren't lost — the problem statement, OSS survey, and resolved gate decisions remain useful reference material when individual primitives or composites land. This LLD becomes the "why we considered this and walked away" artifact.