diff --git a/docs/development.md b/docs/development.md index ccf5747..c32fb56 100644 --- a/docs/development.md +++ b/docs/development.md @@ -4,7 +4,7 @@ ### Prerequisites -- Go 1.25+ +- Go 1.26+ - [Ginkgo CLI](https://onsi.github.io/ginkgo/#installing-ginkgo) (for BDD test runner) ### Install Ginkgo CLI @@ -427,6 +427,7 @@ internal/ # Application packages (not importable externally) wait/ # VMI readiness polling (errgroup) cleanup/ # Label-based teardown (VMs, Services, Secrets) audit/ # SQLite audit tracking (Auditor interface, schema, records) + orchestrator/ # Run + Cleanup orchestrators (errgroup concurrency, VM creation flow) workloads/ # Workload + MultiVMWorkload interfaces, nine implementations, registry testutil/ # Shared test helpers for integration and E2E tests tests/ # Tests requiring external infrastructure @@ -452,14 +453,14 @@ The codebase follows a strict layered architecture where each layer depends only | Layer | Packages | Goroutines | Purpose | |-------|----------|------------|---------| | 0 | `constants` | No | Pure values — API coordinates, labels, defaults | -| 1 | `config`, `cloudinit`, `cluster`, `logging` | No | Configuration, cloud-init YAML, K8s client init, structured logging | +| 1 | `config`, `cloudinit`, `cluster`, `logging`, `audit` | No | Configuration, cloud-init YAML, K8s client init, structured logging, audit tracking | | 2 | `vm`, `resources`, `wait` | Yes | K8s CRUD operations with retry, readiness polling | | 3 | `workloads` | No | Pure data producers — cloud-init specs, resource structs | -| 4 | `cmd/virtwork`, `cleanup`, `audit` | Yes | Orchestration, teardown, and audit tracking | +| 4 | `cmd/virtwork`, `orchestrator`, `cleanup` | Yes | Dependency wiring, orchestration, teardown | ### Concurrency Pattern -Go's native concurrency is used throughout. Parallel operations (VM creation, readiness polling, cleanup) use `errgroup.Group` for structured error handling, with `context.Context` for timeouts and cancellation. +Go's native concurrency is used throughout. The `internal/orchestrator` package drives parallel VM creation and readiness polling via `errgroup.Group` for structured error handling, with `context.Context` for timeouts and cancellation. ```go g, ctx := errgroup.WithContext(ctx) @@ -517,9 +518,9 @@ func (w *MyWorkload) CloudInitUserdata() (string, error) { |--------|---------|---------------| | `ExtraVolumes()` | `nil` | VM needs additional volume mounts | | `ExtraDisks()` | `nil` | VM needs additional disk definitions | -| `DataVolumeTemplates()` | `nil` | Workload needs persistent storage | +| `DataVolumeTemplates()` | `nil, nil` | Workload needs persistent storage (returns `[]DataVolumeTemplateSpec, error`) | | `RequiresService()` | `false` | VMs need a K8s Service for communication | -| `ServiceSpec(namespace)` | `nil` | Define the Service when `RequiresService()` is true | +| `ServiceSpec()` | `nil` | Define the Service when `RequiresService()` is true | | `VMCount()` | `1` | Workload needs multiple VMs (e.g., server/client) | For multi-VM workloads with per-role userdata, implement the `MultiVMWorkload` interface: @@ -571,17 +572,22 @@ If your workload needs more than one role of VM (e.g., a server and a client), i // In internal/workloads/workload.go type MultiVMWorkload interface { Workload - Roles() []string + RoleDistribution() []RoleSpec UserdataForRole(role string, namespace string) (string, error) } + +type RoleSpec struct { + Role string + VMCount int +} ``` Pattern: 1. Embed `BaseWorkload` and store any per-workload state (e.g., a `Namespace` field for in-cluster DNS). -2. Override `VMCount()` to return `count * len(Roles())` so the orchestrator creates the right number of VMs per role. -3. Override `Roles()` to return the role names (e.g., `[]string{"server", "client"}`). -4. Override `UserdataForRole(role, namespace)` to return role-specific cloud-init. The default `CloudInitUserdata()` typically delegates to `UserdataForRole("server", namespace)`. +2. Implement `RoleDistribution()` to return per-role VM counts (e.g., `[]RoleSpec{{Role: "server", VMCount: 1}, {Role: "client", VMCount: 1}}`). +3. Override `VMCount()` to sum the `RoleSpec.VMCount` values from `RoleDistribution()`. +4. Implement `UserdataForRole(role, namespace)` to return role-specific cloud-init. The default `CloudInitUserdata()` typically delegates to `UserdataForRole("server", namespace)`. 5. Set `RequiresService()` to `true` and provide a `ServiceSpec()` selecting server VMs by the `virtwork/role: server` label that the orchestrator applies automatically. 6. Clients reach servers via the in-cluster DNS name `..svc.cluster.local` — never poll for pod IPs. @@ -594,7 +600,7 @@ If your workload needs persistent storage inside the VM: 1. Override `DataVolumeTemplates()` to return a CDI `DataVolume` for each volume needed. Use `vm.BuildDataVolumeTemplate(name, size)` from `internal/vm`. 2. Override `ExtraVolumes()` and `ExtraDisks()` to wire the DataVolume into the VM. **Always set the `Serial` field on the `Disk`** — the in-VM script discovers the device through `/dev/disk/by-id/virtio-`, which is deterministic across reboots (unlike `/dev/vdX`, which is not). 3. In your cloud-init userdata, write the shared `diskSetupScript(serial, mountPoint)` helper (from `internal/workloads/workload.go`) as the first script. It waits for the symlink, formats with XFS if empty, mounts, and writes `/etc/fstab` for persistence across reboots. -4. The orchestrator's `namespaceDataVolumes` helper automatically suffixes DV template names with the VM name to avoid collisions across multiple VMs of the same workload. Your template name should be the un-suffixed base (e.g., `virtwork-chaos-disk-data`). +4. The `NamespaceDataVolumes` helper in `internal/orchestrator/types.go` automatically suffixes DV template names with the VM name to avoid collisions across multiple VMs of the same workload. Your template name should be the un-suffixed base (e.g., `virtwork-chaos-disk-data`). Reference workloads: `disk.go` (single fio mount), `database.go` (PostgreSQL data dir), `chaos_disk.go` (fill/release loop). All three follow the same pattern.