diff --git a/CLAUDE.md b/CLAUDE.md index d17099a..fb5da43 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,6 +6,7 @@ - Use conventional commit messages (feat:, fix:, test:, docs:, refactor:, chore:) - DO NOT commit secrets or other sensitive information - DO NOT commit engineering journals +- Every commit must include a DCO `Signed-off-by` trailer (enforced by `commit-msg` hook); do NOT add a `Co-Authored-By` trailer ## Role Context - Expert Go developer @@ -17,6 +18,7 @@ - BDD: Use Ginkgo BDD framework - Use goroutines and channels for concurrency - Use mermaid for documentation diagrams +- Use `internal/logging.NewLogger()` for structured logging in any code that performs I/O; do not use `fmt.Fprintf` or `log.Printf`. The pure data layer (`internal/workloads`, `internal/cloudinit`) should not log. ## Testing - Unit tests: `{source}_test.go` files alongside source @@ -27,4 +29,11 @@ ## Database - When a database is needed, use SQLite for local testing - Strictly adhere to PostgreSQL syntax standards (e.g., use standard SQL for dates, avoid loose typing) -- Assume the production DB is strict. \ No newline at end of file +- Assume the production DB is strict. + +## Documentation Conventions +- `docs/README.md` is the index/TOC for the `docs/` directory; link new docs from there. +- Use mermaid for architecture, flow, sequence, ERD, and topology diagrams. +- Do NOT reference bare issue numbers (`#123`) in published docs — if context from an issue matters, fold the relevant information into the doc prose so readers don't need to chase the tracker. +- Two docs are intentionally frozen as historical snapshots and should not be updated piecemeal: `docs/implementation-plan.md` and `docs/openshift-virtualization-workload-automation.md`. Add new content to the live docs instead (architecture.md, development.md, configuration.md, etc.). +- Engineering journals are not committed to this repo — that is the policy, not a missing directory. \ No newline at end of file diff --git a/README.md b/README.md index 8212049..86cc688 100644 --- a/README.md +++ b/README.md @@ -100,13 +100,31 @@ virtwork cleanup --run-id ## Workloads +virtwork ships nine built-in workloads, grouped by purpose. With `--vm-count 1` and no `--workloads` filter, a full run creates **11 VMs**: seven single-VM workloads plus two server/client pairs (network and tps). + +**Core load generators** — saturate one resource per VM: + | Workload | VMs | Description | Tools | |----------|-----|-------------|-------| -| **cpu** | N (configurable) | Continuous CPU stress | `stress-ng --cpu 0 --cpu-method all` | -| **memory** | N (configurable) | Memory pressure at 80% | `stress-ng --vm 1 --vm-bytes 80%` | -| **database** | N (configurable) | PostgreSQL with pgbench loop | `pgbench -c 10 -j 2 -T 300` | -| **network** | N×2 (server + client pairs) | Bidirectional throughput | `iperf3 --bidir` | -| **disk** | N (configurable) | Mixed random and sequential I/O | `fio` with multiple profiles | +| **cpu** | N | Continuous CPU stress | `stress-ng --cpu 0 --cpu-method all` | +| **memory** | N | Memory pressure at 80% | `stress-ng --vm 1 --vm-bytes 80%` | +| **disk** | N | Mixed random and sequential I/O on a data disk | `fio` with multiple profiles | +| **database** | N | PostgreSQL with pgbench loop on a data disk | `pgbench -c 10 -j 2 -T 300` | + +**Multi-VM benchmarks** — server/client pairs coordinated via a ClusterIP Service: + +| Workload | VMs | Description | Tools | +|----------|-----|-------------|-------| +| **network** | N × 2 | Bidirectional throughput | `iperf3 --bidir` on port 5201 | +| **tps** | N × 2 | TCP request/response + HTTP file transfer | `netperf` (12865/12866) + Python HTTP server (8080) | + +**Chaos engineering** — inject failures inside the VM to test resilience. ⚠️ See [docs/chaos-workloads.md](docs/chaos-workloads.md) before deploying: + +| Workload | VMs | Description | Tools | +|----------|-----|-------------|-------| +| **chaos-disk** | N | Fill the data disk to a target percentage, release, repeat | `fallocate`, `dd` | +| **chaos-network** | N | Inject latency and packet loss on egress | `tc` + `netem` | +| **chaos-process** | N | Randomly send signals to non-essential processes | shell + `ps`/`kill` | All workloads run as systemd services inside the VMs, surviving reboots and auto-restarting on failure. @@ -118,7 +136,7 @@ Deploy VMs with workloads. ``` Flags: - --workloads strings Workloads to deploy (default [cpu,database,disk,memory,network]) + --workloads strings Workloads to deploy (default: all nine — chaos-disk, chaos-network, chaos-process, cpu, database, disk, memory, network, tps) --vm-count int Number of VMs per workload (default 1) --cpu-cores int CPU cores per VM --memory string Memory per VM (e.g., 2Gi) @@ -163,6 +181,8 @@ virtwork uses a priority chain for configuration (highest to lowest): 3. YAML config file (`--config`) 4. Defaults +The tables below cover the common surface. For a complete reference of every flag, environment variable, YAML key, and per-workload parameter, see [docs/configuration.md](docs/configuration.md). + ### Environment Variables | Variable | Description | @@ -205,6 +225,8 @@ The audit database records execution parameters, timestamps, workload details, V No SSH credentials are stored — only a boolean indicating whether SSH authentication was configured. +For the full schema (five tables with relationships and column descriptions) and common query patterns, see [docs/audit-schema.md](docs/audit-schema.md). + ```bash # Disable audit tracking virtwork run --no-audit @@ -246,6 +268,8 @@ When no SSH flags are provided, no user account is configured in the VMs. virtwork can run as a pod on the cluster using the provided Kustomize manifests in `deploy/`. The deployment manifest uses a semantic version tag (e.g., `v0.0.1`) to pin the image for reproducible deployments. Update the version tag in `deploy/deployment.yaml` when upgrading. For development or testing, you can use `quay.io/opdev/virtwork:latest`. +The section below is a quick reference. For a manifest-by-manifest deep-dive — resource topology, RBAC scope, image pinning policy, sizing, audit-DB persistence — see [docs/deployment.md](docs/deployment.md). + ### Deploy with Kustomize ```bash @@ -293,9 +317,11 @@ The codebase follows a strict layered architecture where each layer depends only ``` Layer 4 — Orchestration cmd/virtwork, cleanup, audit -Layer 3 — Workload Defs workloads (interface, cpu, memory, database, network, disk, registry) +Layer 3 — Workload Defs workloads (Workload + MultiVMWorkload interfaces, registry, + cpu, memory, disk, database, network, tps, chaos-disk, + chaos-network, chaos-process) Layer 2 — K8s Abstractions vm, resources, wait -Layer 1 — Infrastructure config, cluster, cloudinit +Layer 1 — Infrastructure config, cluster, cloudinit, logging Layer 0 — Definitions constants ``` @@ -313,15 +339,18 @@ virtwork/ │ ├── config/ # Viper-based config priority chain │ ├── cluster/ # controller-runtime client init │ ├── cloudinit/ # Cloud-config YAML builder +│ ├── logging/ # Structured logger (log/slog wrapper) │ ├── vm/ # VM spec construction + CRUD + retry │ ├── resources/ # Namespace + Service + Secret helpers │ ├── wait/ # VMI readiness polling │ ├── cleanup/ # Label-based teardown (VMs, Services, Secrets) │ ├── audit/ # SQLite audit tracking (Auditor interface, schema, records) -│ ├── workloads/ # Workload interface + 5 implementations + registry +│ ├── workloads/ # Workload + MultiVMWorkload interfaces, 9 implementations, registry │ └── testutil/ # Shared test helpers for integration + E2E ├── tests/ │ └── e2e/ # E2E acceptance tests (//go:build e2e) +├── build/ +│ └── golden-image/ # Optional Fedora container disk with pre-installed tools ├── deploy/ # Kustomize manifests for OpenShift deployment │ ├── kustomization.yaml │ ├── namespace.yaml @@ -332,11 +361,22 @@ virtwork/ │ ├── pvc.yaml │ └── deployment.yaml ├── Dockerfile # Multi-stage build (Debian builder + UBI9 runtime) +├── Dockerfile.ci # CI variant of the runtime image ├── entrypoint.sh # Container entrypoint (auto-run or sleep) +├── Makefile # CI targets (test, vet, lint, build, ci, verify) ├── docs/ +│ ├── README.md # Documentation index │ ├── architecture.md # Layered architecture and diagrams │ ├── development.md # Developer guide -│ └── implementation-plan.md # Phased build plan +│ ├── configuration.md # Complete config reference (flags, env, YAML) +│ ├── deployment.md # OpenShift deployment deep-dive +│ ├── audit-schema.md # SQLite audit schema reference +│ ├── chaos-workloads.md # Chaos engineering workload guide +│ ├── virtwork-vs-kube-burner.md # Positioning vs kube-burner +│ ├── guide/ # Hands-on guides (overview, deploying, adding workloads) +│ ├── implementation-plan.md # Historical: original phased build plan +│ └── openshift-virtualization-workload-automation.md # Historical: original design rationale +├── OWNERS ├── go.mod └── go.sum ``` @@ -371,6 +411,17 @@ go test -tags "integration e2e" ./... go build -o virtwork ./cmd/virtwork ``` +### Makefile shortcuts + +```bash +make help # list all targets +make test # unit tests with race detector and coverage +make ci # vet + test + build (no cluster required) +make verify # fmt + vet + lint + test (full pre-commit) +make build # build binary to bin/virtwork +make container-build # build the OCI image locally +``` + See [docs/development.md](docs/development.md) for the full developer guide, including instructions for adding new workloads. ## License diff --git a/build/golden-image/README.md b/build/golden-image/README.md index 7161bea..7685fa3 100644 --- a/build/golden-image/README.md +++ b/build/golden-image/README.md @@ -11,11 +11,16 @@ The golden image is based on `quay.io/containerdisks/fedora:41` and includes all - **stress-ng** — CPU and memory stress testing (cpu, memory workloads) - **fio** — Flexible I/O tester for disk benchmarking (disk workload) - **iperf3** — Network performance testing (network workload) +- **netperf** — TCP_RR transaction performance (tps workload) +- **python3** — HTTP file server for the tps workload's application-layer transfers - **postgresql-server** — PostgreSQL database with pgbench (database workload) -- **iproute-tc** — Traffic control for network chaos engineering (future chaos workloads) -- **iptables-nft** — Firewall rules for network partition simulation (future chaos workloads) +- **procps-ng** — `ps`, `pkill`, `kill` for chaos-process +- **iproute-tc** — Traffic control (`tc`) and the `sch_netem` kernel module hooks used by the chaos-network workload to inject latency and packet loss +- **iptables-nft** — Firewall rules; reserved for future network partition / blackhole scenarios -Additional tools like `fallocate`, `dd`, `kill`, and `pkill` are already present in the base Fedora image. +Additional tools like `fallocate` and `dd` (used by chaos-disk) are already present in the base Fedora image. + +Workloads that need persistent storage (disk, database, chaos-disk) discover their data volume through `/dev/disk/by-id/virtio-` using the shared `diskSetupScript` helper — they do not depend on any tools beyond the standard userspace utilities already in the base image (`blkid`, `mkfs.xfs`, `mount`, `readlink`). ## Building @@ -131,11 +136,17 @@ The current approach is simpler and maintains backward compatibility. ## Future Enhancements 1. **Multi-architecture support**: Build for arm64 in addition to amd64 -2. **Automated builds**: GitHub Actions workflow on schedule +2. **Automated builds**: CI workflow on schedule 3. **Image scanning**: Add Trivy security scanning 4. **Semantic versioning**: Pin specific package versions for reproducibility 5. **Image variants**: Create minimal/full variants for different use cases +## Related Docs + +- [docs/chaos-workloads.md](../../docs/chaos-workloads.md) — operator guide for the chaos workloads that use `iproute-tc`, `procps-ng`, and `fallocate` +- [docs/deployment.md](../../docs/deployment.md) — how to set the golden image as the default container disk in your deployment +- [docs/configuration.md](../../docs/configuration.md) — `--container-disk-image` flag, `VIRTWORK_CONTAINER_DISK_IMAGE` env var, and `container_disk_image` YAML key + ## License Apache License 2.0. See [LICENSE](../../LICENSE). diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..934df8c --- /dev/null +++ b/docs/README.md @@ -0,0 +1,46 @@ +# Virtwork Documentation + +This is the documentation index for virtwork. Start with the top-level [README](../README.md) for project orientation and quick start; come here when you need a specific reference. + +## Start Here + +- [../README.md](../README.md) — Project overview, motivation, installation, quick start, CLI reference, configuration summary, OpenShift deployment overview, architecture summary +- [architecture.md](architecture.md) — Layered architecture, mermaid diagrams, concurrency model, workload class diagram, key design decisions + +## Guides + +Hands-on, narrative walkthroughs for first-time users and new contributors. + +- [guide/01-overview.md](guide/01-overview.md) — Mental model: the deploy-and-exit lifecycle, the `Workload` and `MultiVMWorkload` interfaces, where to find things in the code +- [guide/02-deploying-workloads.md](guide/02-deploying-workloads.md) — Nine scenarios from dry-run to chaos to cluster-side deployment, with copy-pasteable commands and expected output +- [guide/03-adding-a-workload.md](guide/03-adding-a-workload.md) — End-to-end TDD walkthrough that builds a new workload from scratch; covers simple, storage-backed, and multi-VM patterns + +## Reference + +Targeted references — read when you need a specific fact. + +- [configuration.md](configuration.md) — Complete reference for every CLI flag, environment variable, YAML key, and ConfigMap key (including per-workload parameters) +- [chaos-workloads.md](chaos-workloads.md) — Operator guide for the three chaos engineering workloads (chaos-disk, chaos-network, chaos-process), including destructive-behavior warnings +- [audit-schema.md](audit-schema.md) — SQLite audit database: five-table ERD, column-by-column reference, common queries, record lifecycle +- [deployment.md](deployment.md) — OpenShift Kustomize deployment deep-dive: resource topology, RBAC scope, ConfigMap/Secret keys, image pinning, sizing, audit-DB persistence +- [virtwork-vs-kube-burner.md](virtwork-vs-kube-burner.md) — Positioning compared to kube-burner; what each tool measures and where they complement each other + +## Contributor + +- [development.md](development.md) — Environment setup, building, running unit/integration/E2E tests, cluster prerequisites, Makefile targets, adding a new workload, SSH/audit configuration, testing patterns, commit conventions +- [../build/golden-image/README.md](../build/golden-image/README.md) — Building and using the optional Fedora-based golden container disk image with pre-installed workload tools + +## Historical + +These documents capture earlier project state and are intentionally not updated. They are preserved as context. + +- [implementation-plan.md](implementation-plan.md) — Original phased build plan (phases 0–12), pre-chaos / pre-tps / pre-logging +- [openshift-virtualization-workload-automation.md](openshift-virtualization-workload-automation.md) — Original design rationale, written before any application code existed + +--- + +> **Conventions for this directory** +> +> - Diagrams are written in mermaid so they render natively on GitHub. Update them in place when the code they describe changes. +> - Bare issue numbers (`#123`) are not used in published docs — any context that matters is folded into the prose. +> - The historical snapshots above are frozen on purpose. New content goes into the live docs, not into them. diff --git a/docs/architecture.md b/docs/architecture.md index 3099637..a586519 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -23,11 +23,16 @@ graph TD subgraph "Layer 3 — Workload Definitions" REGISTRY["internal/workloads/registry.go\nregistry + lookup"] IFACE["internal/workloads/workload.go\nWorkload interface"] + MULTI["internal/workloads/workload.go\nMultiVMWorkload interface\n(Roles, UserdataForRole)"] CPU["internal/workloads/cpu.go\nstress-ng CPU"] MEM["internal/workloads/memory.go\nstress-ng VM memory"] + DISK["internal/workloads/disk.go\nfio profiles"] DB["internal/workloads/database.go\nPostgreSQL + pgbench"] NET["internal/workloads/network.go\niperf3 server/client"] - DISK["internal/workloads/disk.go\nfio profiles"] + TPS["internal/workloads/tps.go\nnetperf TCP_RR + HTTP"] + CDISK["internal/workloads/chaos_disk.go\nfill/release loop"] + CNET["internal/workloads/chaos-network.go\ntc/netem latency + loss"] + CPROC["internal/workloads/chaos_process.go\nrandom signal killer"] end subgraph "Layer 2 — K8s Abstractions" @@ -40,6 +45,7 @@ graph TD CLUSTER["internal/cluster/cluster.go\ncontroller-runtime client init"] CONFIG["internal/config/config.go\nViper config"] CLOUDINIT["internal/cloudinit/cloudinit.go\ncloud-config YAML builder"] + LOGGING["internal/logging/logging.go\nstructured slog logger"] end subgraph "Layer 0 — Definitions" @@ -54,6 +60,7 @@ graph TD CMD --> WAIT CMD --> CLEANUP CMD --> AUDIT + CMD --> LOGGING AUDIT --> CONFIG AUDIT --> CONST @@ -64,15 +71,25 @@ graph TD REGISTRY --> CPU REGISTRY --> MEM + REGISTRY --> DISK REGISTRY --> DB REGISTRY --> NET - REGISTRY --> DISK + REGISTRY --> TPS + REGISTRY --> CDISK + REGISTRY --> CNET + REGISTRY --> CPROC CPU --> IFACE MEM --> IFACE - DB --> IFACE - NET --> IFACE DISK --> IFACE + DB --> IFACE + CDISK --> IFACE + CNET --> IFACE + CPROC --> IFACE + + NET --> MULTI + TPS --> MULTI + MULTI --> IFACE VM --> CLUSTER VM --> CONST @@ -83,6 +100,7 @@ graph TD WAIT --> CLUSTER WAIT --> CONST + WAIT --> LOGGING CLUSTER --> CONST CONFIG --> CONST @@ -150,6 +168,7 @@ graph LR | `internal/config` | No | One-time Viper load at startup | | `internal/cloudinit` | No | Pure string/YAML generation | | `internal/cluster` | No | One-time client init at startup | +| `internal/logging` | No | Stateless `slog` wrapper; safe to share across goroutines that call it | | `internal/vm` | Yes | CRUD operations run in errgroup goroutines; retry loops use `time.Sleep` | | `internal/resources` | Yes | Namespace/Service/Secret creation can run concurrently | | `internal/wait` | Yes | Concurrent VMI polling via errgroup; uses `time.Sleep` between polls | @@ -231,6 +250,12 @@ classDiagram +VMCount() int } + class MultiVMWorkload { + <> + +Roles() []string + +UserdataForRole(role, namespace) (string, error) + } + class BaseWorkload { +Config WorkloadConfig +SSHUser string @@ -249,43 +274,80 @@ classDiagram class CPUWorkload { +Name() "cpu" +CloudInitUserdata() stress-ng --cpu config - +VMResources() cpu/memory from config } class MemoryWorkload { +Name() "memory" +CloudInitUserdata() stress-ng --vm config - +VMResources() cpu/memory from config + } + + class DiskWorkload { + +Name() "disk" + +CloudInitUserdata() fio profiles + +DataVolumeTemplates() blank DV for /mnt/data + +ExtraDisks() data disk with virtio Serial } class DatabaseWorkload { +Name() "database" +CloudInitUserdata() postgresql + pgbench +DataVolumeTemplates() blank DV for /var/lib/pgsql/data + +ExtraDisks() data disk with virtio Serial } class NetworkWorkload { +Namespace string +Name() "network" - +VMCount() count * 2 (server+client pairs) + +VMCount() count * 2 + +Roles() ["server", "client"] + +UserdataForRole(role, ns) iperf3 server or client +RequiresService() true - +ServerUserdata() iperf3 -s - +ClientUserdata() iperf3 -c - +ServiceSpec() ClusterIP for server + +ServiceSpec() ClusterIP virtwork-iperf3-server :5201 } - class DiskWorkload { - +Name() "disk" - +CloudInitUserdata() fio profiles + class TPSWorkload { + +Namespace string + +Name() "tps" + +VMCount() count * 2 + +Roles() ["server", "client"] + +UserdataForRole(role, ns) netperf + HTTP server / client loop + +RequiresService() true + +ServiceSpec() ClusterIP virtwork-tps-server :12865/:12866/:8080 + +Params file-size, iterations, duration + } + + class ChaosDiskWorkload { + +Name() "chaos-disk" + +CloudInitUserdata() fallocate fill / rm release loop +DataVolumeTemplates() blank DV for /mnt/data + +ExtraDisks() data disk with virtio Serial } + class ChaosNetworkWorkload { + +Latency int (ms) + +PacketLoss float64 (%) + +Name() "chaos-network" + +CloudInitUserdata() tc qdisc netem delay + loss + } + + class ChaosProcessWorkload { + +Name() "chaos-process" + +CloudInitUserdata() random kill loop with excluded patterns + } + + Workload <|-- MultiVMWorkload Workload <|.. BaseWorkload BaseWorkload <|-- CPUWorkload BaseWorkload <|-- MemoryWorkload + BaseWorkload <|-- DiskWorkload BaseWorkload <|-- DatabaseWorkload BaseWorkload <|-- NetworkWorkload - BaseWorkload <|-- DiskWorkload + BaseWorkload <|-- TPSWorkload + BaseWorkload <|-- ChaosDiskWorkload + BaseWorkload <|-- ChaosNetworkWorkload + BaseWorkload <|-- ChaosProcessWorkload + MultiVMWorkload <|.. NetworkWorkload + MultiVMWorkload <|.. TPSWorkload ``` `BaseWorkload` is an embedded struct that provides default implementations for optional interface methods. Concrete workloads embed `BaseWorkload` and override only the methods they need — idiomatic Go composition over inheritance. @@ -298,9 +360,13 @@ classDiagram |----------|----------|-------------|-------------|----------|---------------| | CPU | N (configurable) | No | No | stress-ng | `stress-ng --cpu 0 --cpu-method all` | | Memory | N (configurable) | No | No | stress-ng | `stress-ng --vm 1 --vm-bytes 80% --vm-method all` | -| Database | N (configurable) | Yes (`/var/lib/pgsql/data`) | No | postgresql-server | `pgbench -c 10 -j 2 -T 300` loop | -| Network | N×2 (server + client pairs) | No | Yes (ClusterIP) | iperf3 | `iperf3 -s` / `iperf3 -c ... --bidir` | | Disk | N (configurable) | Yes (`/mnt/data`) | No | fio | Mixed R/W + sequential write profiles | +| Database | N (configurable) | Yes (`/var/lib/pgsql/data`) | No | postgresql-server | `pgbench -c 10 -j 2 -T 300` loop | +| Network | N × 2 (server + client) | No | Yes — `virtwork-iperf3-server` :5201 | iperf3 | `iperf3 -s` / `iperf3 -c ... --bidir` | +| TPS | N × 2 (server + client) | No | Yes — `virtwork-tps-server` :12865 / :12866 / :8080 | netperf, python3 | `netperf -t TCP_RR` + curl HTTP file fetch loop | +| Chaos-disk | N (configurable) | Yes (`/mnt/data`) | No | (golden image: `fallocate`, `dd`) | Fill to target percent, sleep, release, repeat | +| Chaos-network | N (configurable) | No | No | iproute-tc (+ `sch_netem` kernel module) | `tc qdisc add ... netem delay 100ms loss 5%` | +| Chaos-process | N (configurable) | No | No | procps-ng | Random `kill -SIGTERM ` of non-essential processes every 30s | --- @@ -407,9 +473,13 @@ Viper's built-in priority chain handles this natively when bound to Cobra flags: | Network VM scaling | `VMCount() = count * 2` | Honors `--vm-count` to create N server/client pairs instead of a single hardcoded pair. | | Cloud-init Secrets | `CloudInitSecretName` → `UserDataSecretRef` | For large userdata, stores cloud-init in a K8s Secret instead of inline in the VM spec. | | Cleanup error semantics | Sequential per-resource deletion with error accumulation | Different from create-time error handling (which is fail-fast). Cleanup continues on individual failures. | -| Audit storage | SQLite (`virtwork.db`) with `Auditor` interface | Local file, zero infrastructure. `NoOpAuditor` when disabled. WAL mode for concurrent safety. | +| Audit storage | SQLite (`virtwork.db`) with `Auditor` interface | Local file, zero infrastructure. `NoOpAuditor` when disabled. WAL mode for concurrent safety. See [audit-schema.md](audit-schema.md) for the full schema. | | Run-to-cleanup linking | `virtwork/run-id` K8s label + `linked_run_ids` JSON array | Labels survive across CLI invocations. JSON array is PostgreSQL JSONB compatible. | | Audit credential policy | No SSH credentials stored | Only `ssh_auth_configured` boolean tracked. Security by design. | +| In-VM disk discovery | `/dev/disk/by-id/virtio-` via the `Serial` field on KubeVirt `Disk` | `/dev/vdX` device ordering is not stable across VM reboots or migrations; the virtio serial provides a deterministic symlink. The shared `diskSetupScript` helper (`internal/workloads/workload.go`) waits for the symlink, formats if empty, mounts, and writes `/etc/fstab`. | +| DataVolume names per VM | DV template names are suffixed with the VM name via `namespaceDataVolumes` (`cmd/virtwork/main.go`) | DataVolume names are namespace-scoped; deploying multiple VMs of the same workload would otherwise collide on the template name. | +| Structured logging | `log/slog` JSON via `internal/logging.NewLogger(out, verbose)` | Machine-parseable logs for pipeline consumption; `--verbose` flips the level from `INFO` to `DEBUG`. New code uses the logger; `fmt.Fprintf` calls were removed from `cmd/virtwork/main.go`. | +| Chaos workload safety | Opt-in by name, namespace-scoped destructive behavior, no platform-level kill switch | Chaos workloads can fill a data PVC, shape egress traffic, or kill processes — all confined to the VM they run in. Namespace isolation is the safety boundary. See [chaos-workloads.md](chaos-workloads.md) for risk and operational guidance. | --- @@ -429,6 +499,8 @@ virtwork/ │ │ └── cluster.go # controller-runtime client init + scheme registration │ ├── cloudinit/ │ │ └── cloudinit.go # Cloud-config YAML builder +│ ├── logging/ +│ │ └── logging.go # Structured slog logger (verbose -> DEBUG) │ ├── vm/ │ │ └── vm.go # VM spec construction + typed CRUD + retry │ ├── resources/ @@ -442,25 +514,43 @@ virtwork/ │ │ ├── schema.go # DDL for 5 audit tables + indexes │ │ └── records.go # WorkloadRecord, VMRecord, ResourceRecord, EventRecord │ ├── workloads/ -│ │ ├── workload.go # Workload interface + BaseWorkload -│ │ ├── registry.go # Registry map + lookup +│ │ ├── workload.go # Workload + MultiVMWorkload interfaces, BaseWorkload, diskSetupScript +│ │ ├── registry.go # Registry map + DefaultRegistry (nine entries) │ │ ├── cpu.go # stress-ng CPU continuous workload │ │ ├── memory.go # stress-ng VM memory pressure workload +│ │ ├── disk.go # fio mixed I/O profiles │ │ ├── database.go # PostgreSQL + pgbench loop -│ │ ├── network.go # iperf3 server/client pair -│ │ └── disk.go # fio mixed I/O profiles +│ │ ├── network.go # iperf3 server/client pair (MultiVMWorkload) +│ │ ├── tps.go # netperf TCP_RR + HTTP file transfer (MultiVMWorkload) +│ │ ├── chaos_disk.go # Fill-and-release disk pressure loop +│ │ ├── chaos-network.go # tc/netem latency + loss injection +│ │ └── chaos_process.go # Random kill loop with excluded process patterns │ └── testutil/ │ ├── testutil.go # Shared test helpers (namespace, connect, cleanup) │ └── binary.go # Binary build/run helpers for E2E ├── tests/ │ └── e2e/ # E2E acceptance tests (//go:build e2e) +├── build/ +│ └── golden-image/ # Optional Fedora container disk with pre-installed tools +├── deploy/ # Kustomize manifests for OpenShift deployment ├── docs/ +│ ├── README.md # Documentation index │ ├── architecture.md # This file │ ├── development.md # Developer guide -│ ├── implementation-plan.md # Phased build plan -│ ├── openshift-virtualization-workload-automation.md # Design plan -│ └── engineering-journals/ # Per-phase development journals +│ ├── configuration.md # Complete config reference +│ ├── deployment.md # OpenShift deployment deep-dive +│ ├── audit-schema.md # SQLite audit schema reference +│ ├── chaos-workloads.md # Chaos engineering workload guide +│ ├── virtwork-vs-kube-burner.md # Positioning vs kube-burner +│ ├── guide/ # Hands-on guides (overview, deploying, adding workloads) +│ ├── implementation-plan.md # Historical: original phased build plan +│ └── openshift-virtualization-workload-automation.md # Historical: original design rationale +├── Dockerfile # Multi-stage build (Debian builder + UBI9 runtime) +├── Dockerfile.ci # CI variant of the runtime image +├── entrypoint.sh +├── Makefile ├── go.mod ├── go.sum +├── OWNERS └── CLAUDE.md ``` diff --git a/docs/audit-schema.md b/docs/audit-schema.md new file mode 100644 index 0000000..4545c9f --- /dev/null +++ b/docs/audit-schema.md @@ -0,0 +1,356 @@ +# Audit Database Schema + +Every `virtwork run` and `virtwork cleanup` execution is recorded in a local SQLite database (`virtwork.db` by default) so that what happened, when, with what configuration, and with what outcome is always recoverable — even when the cluster itself is gone. + +This document is the reference for the schema: tables, columns, indexes, relationships, and how rows are produced over an execution's lifetime. + +## Storage and Implementation + +- **Backing store:** SQLite, opened in WAL mode for concurrent-read safety. The file path is `virtwork.db` by default, overridden via `--audit-db` / `VIRTWORK_AUDIT_DB` / `audit_db` YAML key. When deployed in-cluster the path is `/data/virtwork.db` on the audit PVC. +- **Interface:** `internal/audit.Auditor` with two implementations: `SQLiteAuditor` (the real one) and `NoOpAuditor` (returned when `--no-audit` or `VIRTWORK_AUDIT=false` is set). +- **PostgreSQL compatibility:** all `TEXT` timestamps use ISO 8601 strings; `linked_run_ids` is a JSON array stored as `TEXT` so the same shape migrates cleanly to PostgreSQL JSONB if needed later. +- **DDL location:** `internal/audit/schema.go`. Record structs that map to inserts live in `internal/audit/records.go`. + +## Entity-Relationship Diagram + +```mermaid +erDiagram + audit_log ||--o{ workload_details : "one run has many workloads" + audit_log ||--o{ vm_details : "one run has many VMs" + audit_log ||--o{ resource_details : "one run has many resources" + audit_log ||--o{ events : "one run has many events" + workload_details ||--o{ vm_details : "one workload has many VMs" + workload_details ||--o{ events : "events optionally reference workload" + vm_details ||--o{ events : "events optionally reference VM" + + audit_log { + INTEGER id PK + TEXT run_id "UNIQUE — UUID for the execution" + TEXT linked_run_ids "JSON array, set on cleanup" + TEXT command "run | dry-run | cleanup" + TEXT status "in_progress | success | failed" + TEXT namespace + TEXT workloads_csv + INTEGER total_vm_count + INTEGER dry_run + INTEGER ssh_auth_configured + TEXT started_at + TEXT completed_at + TEXT error_summary + } + + workload_details { + INTEGER id PK + INTEGER audit_id FK + TEXT workload_type "cpu, tps, chaos-disk, ..." + INTEGER vm_count + INTEGER cpu_cores + TEXT memory + INTEGER has_data_disk + INTEGER requires_service + TEXT status "pending | created | failed" + } + + vm_details { + INTEGER id PK + INTEGER audit_id FK + INTEGER workload_id FK + TEXT vm_name + TEXT component + TEXT role "server | client | (empty)" + TEXT phase + TEXT status "planned | created | ready | deleted | failed" + TEXT created_at + TEXT ready_at + TEXT deleted_at + } + + resource_details { + INTEGER id PK + INTEGER audit_id FK + TEXT resource_type "Service | Secret" + TEXT resource_name + TEXT status "created | deleted" + } + + events { + INTEGER id PK + INTEGER audit_id FK + INTEGER vm_id FK + INTEGER workload_id FK + TEXT event_type + TEXT message + TEXT error_detail + TEXT occurred_at + } +``` + +## Tables + +### `audit_log` — one row per execution + +| Column | Type | Notes | +|---|---|---| +| `id` | INTEGER PK | Autoincrement | +| `run_id` | TEXT UNIQUE NOT NULL | UUID applied to all K8s resources via the `virtwork/run-id` label | +| `linked_run_ids` | TEXT | JSON array of run-IDs discovered during cleanup (NULL outside of cleanup) | +| `command` | TEXT NOT NULL | `run`, `dry-run`, `cleanup`, or `cleanup --dry-run` | +| `status` | TEXT NOT NULL | `in_progress` initially, `success` or `failed` on completion | +| `kubeconfig_path` | TEXT | Path used to connect (NULL when in-cluster) | +| `cluster_context` | TEXT | Reserved — currently unused | +| `namespace` | TEXT NOT NULL | Target namespace | +| `container_disk_image` | TEXT | Configured boot image | +| `default_cpu_cores` | INTEGER | Global `--cpu-cores` default | +| `default_memory` | TEXT | Global `--memory` default | +| `data_disk_size` | TEXT | Global `--disk-size` default | +| `workloads_csv` | TEXT | Comma-separated list of workload names requested | +| `total_vm_count` | INTEGER | Total VMs planned across all workloads | +| `total_workload_count` | INTEGER | Number of workload types deployed | +| `dry_run` | INTEGER NOT NULL | 0 or 1 | +| `ssh_auth_configured` | INTEGER NOT NULL | 1 if any SSH credential was provided. **Credentials are never stored.** | +| `cleanup_mode` | TEXT | Reserved — currently unused | +| `wait_for_ready` | INTEGER NOT NULL | 0 or 1; reflects `--no-wait` inverted | +| `ready_timeout_seconds` | INTEGER | Effective readiness timeout | +| `vms_deleted` | INTEGER | Cleanup only: count from `CleanupResult` | +| `services_deleted` | INTEGER | Cleanup only | +| `secrets_deleted` | INTEGER | Cleanup only | +| `namespace_deleted` | INTEGER | Cleanup only: 0 or 1 | +| `started_at` | TEXT NOT NULL | ISO 8601 timestamp | +| `completed_at` | TEXT | NULL while in flight | +| `error_summary` | TEXT | Populated when `status = 'failed'` | + +Indexes: `started_at`, `namespace`, `status`, `run_id`. + +### `workload_details` — one row per workload-type per execution + +| Column | Type | Notes | +|---|---|---| +| `id` | INTEGER PK | | +| `audit_id` | INTEGER NOT NULL | FK → `audit_log.id` | +| `workload_type` | TEXT NOT NULL | `cpu`, `memory`, `disk`, `database`, `network`, `tps`, `chaos-disk`, `chaos-network`, `chaos-process` | +| `enabled` | INTEGER NOT NULL | 0 or 1 (always 1 in current orchestration) | +| `vm_count` | INTEGER NOT NULL | Reported by `Workload.VMCount()` — `N` for single-VM, `N × len(Roles())` for multi-VM | +| `cpu_cores` | INTEGER NOT NULL | Effective per-VM CPU cores | +| `memory` | TEXT NOT NULL | Effective per-VM memory (e.g., `2Gi`) | +| `has_data_disk` | INTEGER NOT NULL | 1 when `DataVolumeTemplates()` is non-empty | +| `data_disk_size` | TEXT | NULL when `has_data_disk = 0` | +| `requires_service` | INTEGER NOT NULL | 1 for multi-VM workloads (network, tps) | +| `status` | TEXT NOT NULL | `pending` → `created` (after VM create succeeds) → `failed` | +| `started_at` | TEXT | Reserved — currently unset | +| `completed_at` | TEXT | Reserved — currently unset | + +Indexes: `audit_id`, `workload_type`. + +### `vm_details` — one row per VM + +| Column | Type | Notes | +|---|---|---| +| `id` | INTEGER PK | | +| `audit_id` | INTEGER NOT NULL | FK → `audit_log.id` | +| `workload_id` | INTEGER | FK → `workload_details.id` | +| `vm_name` | TEXT NOT NULL | e.g., `virtwork-cpu-0`, `virtwork-network-server-0` | +| `namespace` | TEXT NOT NULL | | +| `component` | TEXT NOT NULL | Workload name (matches `workload_details.workload_type`) | +| `role` | TEXT | `server` / `client` for multi-VM workloads; empty for single-VM | +| `cpu_cores` | INTEGER NOT NULL | | +| `memory` | TEXT NOT NULL | | +| `container_disk_image` | TEXT NOT NULL | | +| `has_data_disk` | INTEGER NOT NULL | | +| `data_disk_size` | TEXT | | +| `phase` | TEXT | Latest known VMI phase (`Pending`, `Scheduled`, `Running`, …) | +| `status` | TEXT NOT NULL | `planned` → `created` → `ready` / `failed` / `deleted` | +| `created_at` | TEXT | Set when `CreateVM` succeeds | +| `ready_at` | TEXT | Set when readiness polling succeeds | +| `deleted_at` | TEXT | Set during cleanup | + +Indexes: `audit_id`, `workload_id`, `vm_name`. + +### `resource_details` — one row per non-VM K8s resource + +Tracks Services and Secrets created by orchestration. (VMs go in `vm_details`.) + +| Column | Type | Notes | +|---|---|---| +| `id` | INTEGER PK | | +| `audit_id` | INTEGER NOT NULL | FK → `audit_log.id` | +| `resource_type` | TEXT NOT NULL | `Service` or `Secret` | +| `resource_name` | TEXT NOT NULL | e.g., `virtwork-iperf3-server`, `virtwork-cpu-0-cloudinit` | +| `namespace` | TEXT NOT NULL | | +| `status` | TEXT NOT NULL | `created` or `deleted` | +| `created_at` | TEXT | | +| `deleted_at` | TEXT | Set during cleanup | + +Indexes: `audit_id`, `resource_type`. + +### `events` — append-only event log + +| Column | Type | Notes | +|---|---|---| +| `id` | INTEGER PK | | +| `audit_id` | INTEGER NOT NULL | FK → `audit_log.id` | +| `vm_id` | INTEGER | FK → `vm_details.id` (optional) | +| `workload_id` | INTEGER | FK → `workload_details.id` (optional) | +| `event_type` | TEXT NOT NULL | See enumeration below | +| `message` | TEXT | Free-form | +| `error_detail` | TEXT | Set when the event represents a failure | +| `occurred_at` | TEXT NOT NULL | | + +Indexes: `audit_id`, `event_type`, `occurred_at`. + +**Common `event_type` values emitted by the orchestrator today:** + +| `event_type` | When emitted | +|---|---| +| `execution_started` | Two per run — once for "Starting" and once for "Planned N VMs across M workloads" | +| `service_created` | After each Service is created | +| `vm_created` | After each VM is successfully created on the cluster | +| `vm_failed` | When `CreateVM` returns an error | +| `vm_ready` | After readiness polling confirms `Running` | +| `vm_timeout` | When a VM fails readiness check | +| `cleanup_started` | Beginning of `cleanup` execution | +| `cleanup_completed` | End of `cleanup` execution, with the deletion counts | + +--- + +## Lifecycle: How Rows Appear + +```mermaid +sequenceDiagram + participant CLI as cmd/virtwork + participant A as Auditor + participant DB as SQLite + + CLI->>A: StartExecution(ctx, "run", cfg) + A->>DB: INSERT audit_log (run_id=UUID, status=in_progress, started_at=now) + A-->>CLI: (execID, runID) + + loop per workload + CLI->>A: RecordWorkload(execID, WorkloadRecord) + A->>DB: INSERT workload_details (audit_id=execID, ...) + A-->>CLI: workloadID + end + + loop per Service to create + CLI->>A: RecordResource(execID, ResourceRecord) + A->>DB: INSERT resource_details + CLI->>A: RecordEvent(execID, event_type=service_created) + A->>DB: INSERT events + end + + loop per VM to create (errgroup) + CLI->>A: RecordVM(execID, workloadID, VMRecord) + A->>DB: INSERT vm_details (status=planned) + CLI->>A: RecordEvent(execID, event_type=vm_created) + A->>DB: INSERT events + end + + Note over CLI: WaitForAllVMsReady ... + + loop per VM ready/timeout + CLI->>A: RecordEvent(execID, event_type=vm_ready|vm_timeout) + A->>DB: INSERT events + end + + CLI->>A: UpdateWorkloadStatus(workloadID, "created") + A->>DB: UPDATE workload_details SET status='created' + + CLI->>A: CompleteExecution(execID, "success", "") + A->>DB: UPDATE audit_log SET status='success', completed_at=now +``` + +Cleanup is similar: a new `audit_log` row is created with `command='cleanup'`, resources are discovered by label, deleted, counted, and the `linked_run_ids` column is filled with the unique `virtwork/run-id` values that were present on the deleted resources. + +--- + +## Common Queries + +### Recent executions + +```sql +SELECT id, run_id, command, status, namespace, total_vm_count, started_at +FROM audit_log +ORDER BY id DESC +LIMIT 10; +``` + +### VMs from a specific run + +```sql +SELECT vm_name, component, role, cpu_cores, memory, status, ready_at +FROM vm_details +WHERE audit_id = (SELECT id FROM audit_log WHERE run_id = ''); +``` + +### Events timeline for a run + +```sql +SELECT occurred_at, event_type, message +FROM events +WHERE audit_id = (SELECT id FROM audit_log WHERE run_id = '') +ORDER BY occurred_at; +``` + +### Failed runs in the last day + +```sql +SELECT id, run_id, command, error_summary, started_at, completed_at +FROM audit_log +WHERE status = 'failed' AND started_at >= datetime('now', '-1 day') +ORDER BY started_at DESC; +``` + +### Which cleanups touched a given run-id? + +```sql +SELECT id, run_id, started_at, vms_deleted, services_deleted, secrets_deleted, linked_run_ids +FROM audit_log +WHERE command LIKE 'cleanup%' AND linked_run_ids LIKE '%%'; +``` + +### How many VMs per workload across all runs? + +```sql +SELECT workload_type, SUM(vm_count) AS total_vms, COUNT(*) AS runs +FROM workload_details +GROUP BY workload_type +ORDER BY total_vms DESC; +``` + +### Average VM-ready latency per workload type + +```sql +SELECT + component, + AVG((julianday(ready_at) - julianday(created_at)) * 86400) AS avg_ready_seconds +FROM vm_details +WHERE ready_at IS NOT NULL AND created_at IS NOT NULL +GROUP BY component +ORDER BY avg_ready_seconds DESC; +``` + +--- + +## For Contributors + +- DDL: `internal/audit/schema.go` (`schemaSQL` constant, executed once on `NewSQLiteAuditor`). +- Insert/update methods: `internal/audit/audit.go` (`SQLiteAuditor` methods). +- Record structs that map to inserts: `internal/audit/records.go` (`WorkloadRecord`, `VMRecord`, `ResourceRecord`, `EventRecord`). +- `NoOpAuditor`: returned when audit is disabled. Implements the same interface with empty methods so call sites never need to nil-check. + +### Adding a new column + +1. Add the column to `schemaSQL` in `schema.go`. SQLite's `CREATE TABLE IF NOT EXISTS` will not alter an existing table — for production-like upgrades, add an explicit `ALTER TABLE` migration. +2. Add the field to the relevant `Record` struct in `records.go`. +3. Update the insert/update method in `audit.go` to include the new column. +4. Update the relevant table section in this document. +5. PostgreSQL compatibility: keep timestamps as ISO 8601 `TEXT`; use `TEXT` for JSON-shaped data (it maps cleanly to JSONB). + +### Adding a new event type + +Just call `RecordEvent(ctx, execID, EventRecord{EventType: "your_new_type", Message: "..."})` from the orchestrator. Add a row to the "Common `event_type` values" table above so the catalog stays current. + +## Related Docs + +- [configuration.md](configuration.md) — audit-related flags, env vars, YAML keys +- [deployment.md](deployment.md) — audit-DB PVC mount at `/data` for in-cluster deployments +- [development.md](development.md) — audit configuration section in the developer guide diff --git a/docs/chaos-workloads.md b/docs/chaos-workloads.md new file mode 100644 index 0000000..a0f8af7 --- /dev/null +++ b/docs/chaos-workloads.md @@ -0,0 +1,281 @@ +# Chaos Workloads + +Virtwork ships three chaos engineering workloads that inject failures inside a VM so that monitoring stacks, alerting rules, application resilience patterns, and recovery procedures can be exercised against realistic — not synthetic — disturbances. + +| Workload | What it does | Tool(s) | Storage | +|---|---|---|---| +| **chaos-disk** | Fills the data disk to a target percentage, sleeps, releases, repeats | `fallocate`, `dd` | DataVolume mounted at `/mnt/data` | +| **chaos-network** | Adds latency and packet loss on the VM's egress interface | `tc` + `netem` | None | +| **chaos-process** | Randomly signals non-essential processes inside the VM | shell + `ps`/`kill` | None | + +All three run as systemd services (`virtwork-chaos-disk.service`, `virtwork-chaos-network.service`, `virtwork-chaos-process.service`) and restart automatically on failure or VM reboot. + +--- + +## ⚠️ Destructive-Behavior Warning + +Chaos workloads are **deliberately destructive within their VM**: + +- **chaos-disk** consumes nearly all of the data PVC and releases it on a loop. While it's in the "fill" phase, any other process on the VM that writes to `/mnt/data` will see `ENOSPC`. +- **chaos-network** rewrites the egress qdisc on the default-route interface. All outbound traffic from that VM — including health checks, metric scrapes, and SSH responses — incurs the configured latency and drop rate. +- **chaos-process** sends signals (default `SIGTERM`) to randomly selected processes every 30 seconds. Long-running tools that are not in the exclusion list can be killed at any time. + +The destructive effects are **confined to the VM** the workload runs in. Virtwork does not have a platform-level kill switch. Use namespace isolation as your safety boundary: deploy chaos workloads in their own namespace, or at least never alongside workloads you care about. + +```bash +# RECOMMENDED: dedicated chaos namespace +virtwork run --namespace virtwork-chaos --workloads chaos-disk,chaos-network,chaos-process + +# DO NOT: mix with workloads you intend to observe +virtwork run --workloads cpu,database,chaos-disk # database and chaos-disk share the namespace +``` + +Cleanup is unchanged from any other workload — `virtwork cleanup` deletes the VMs and the chaos behavior stops with them. + +--- + +## chaos-disk + +### What it does + +Continuously fills the mounted data disk to a configurable percentage, holds the fill for a configurable duration, then releases it. This produces sustained disk-pressure events that exercise: + +- Filesystem-fill alerts in your monitoring stack +- Eviction or pause behavior in components that watch disk usage +- Recovery paths that depend on freeing disk space + +### Flow + +```mermaid +flowchart LR + A[Compute target_kb
= total * fill-percent] --> B{fill_kb > 0?} + B -->|yes| C[fallocate or dd
chaos-disk-fill file] + B -->|no| D[Skip fill] + C --> E[Sleep fill-sleep
default 60s] + D --> E + E --> F[rm chaos-disk-fill] + F --> G[Sleep release-sleep
default 30s] + G --> A +``` + +### Configuration + +| YAML key (`workloads.chaos-disk.params.*`) | Default | Effect | +|---|---|---| +| `mount` | `/mnt/data` | Mountpoint of the data disk to fill | +| `fill-percent` | `90` | Target fill percentage of the mountpoint | +| `fill-sleep` | `60` | Seconds to hold the fill before releasing | +| `release-sleep` | `30` | Seconds to wait empty before refilling | + +Example: + +```yaml +workloads: + chaos-disk: + enabled: true + vm_count: 1 + cpu_cores: 1 + memory: 1Gi + params: + fill-percent: "80" + fill-sleep: "120" + release-sleep: "60" +``` + +The data disk size comes from `--disk-size` / `VIRTWORK_DATA_DISK_SIZE` / `data_disk_size` (default `10Gi`). The disk is provisioned as a CDI `DataVolume`, mounted in-VM via the virtio Serial pattern (`/dev/disk/by-id/virtio-virtwork-chdisk`), formatted XFS, and persisted in `/etc/fstab` by the shared `diskSetupScript` helper. + +### Example + +```bash +virtwork run --workloads chaos-disk --disk-size 5Gi --ssh-user virtwork --ssh-key-file ~/.ssh/id_ed25519.pub + +# Watch the fill/release cycle from inside the VM: +virtctl ssh --ssh-key ~/.ssh/id_ed25519 virtwork@virtwork-chaos-disk-0 -n virtwork +watch -n 5 'df -h /mnt/data' +``` + +You should see `/mnt/data` use rise to ~90%, hold for ~60s, drop back near baseline, hold for ~30s, and repeat. + +### What to observe + +- `kubelet_volume_stats_used_bytes{persistentvolumeclaim=~"virtwork-chaos-disk.*"}` from kube-state-metrics +- Pod or VMI events around disk pressure +- Any application metrics tied to free-space thresholds you have configured + +--- + +## chaos-network + +### What it does + +Applies a `tc` qdisc with `netem` to the VM's default-route interface, adding latency and packet loss to every egress packet. The qdisc is installed once on service start and removed cleanly on service stop, so a `systemctl restart virtwork-chaos-network` from inside the VM disables and re-enables the impairment. + +### Configuration + +| YAML key (`workloads.chaos-network.params.*`) | Default | Effect | +|---|---|---| +| `latency-ms` | `100` | One-way latency to inject (`netem delay ms`) | +| `packet-loss-percent` | `5.0` | Packet drop rate (`netem loss %`) | + +Example: + +```yaml +workloads: + chaos-network: + enabled: true + vm_count: 1 + cpu_cores: 1 + memory: 1Gi + params: + latency-ms: "250" + packet-loss-percent: "10" +``` + +### Dependencies + +The workload installs `iproute-tc` and loads the `sch_netem` kernel module on start. The base Fedora container disk has both available; the golden image (`build/golden-image/`) pre-installs `iproute-tc` to avoid the first-boot package install. `sch_netem` is loaded from the `kernel-modules-extra` package, which the cloud-init runs `dnf install` for on first boot. + +### Example + +```bash +virtwork run --workloads chaos-network --ssh-user virtwork --ssh-key-file ~/.ssh/id_ed25519.pub + +# Inside the VM, confirm the qdisc is active: +virtctl ssh --ssh-key ~/.ssh/id_ed25519 virtwork@virtwork-chaos-network-0 -n virtwork +tc qdisc show dev eth0 +# qdisc netem 8001: root refcnt 2 limit 1000 delay 100ms loss 5% + +ping -c 10 1.1.1.1 +# Expect ~100ms RTT and occasional packet loss +``` + +### What to observe + +- Increased latency in your service-to-service traces (when chaos-network sits on the client side) +- Retry / circuit-breaker behavior in HTTP clients +- TCP retransmit counters (`netstat -s` inside the VM, or node-level metrics if you scrape them) + +--- + +## chaos-process + +### What it does + +Every 30 seconds (configurable), randomly selects one process from the VM that is **not** in the exclusion list and sends it `SIGTERM` (configurable). Processes are eligible when their PID is at or above the configured minimum (default 1000) — this is a coarse filter to leave kernel threads and core systemd-managed processes alone. + +### Exclusion list + +The following process-name patterns are never targeted (built into the script): + +- `systemd` +- `sshd` +- `dbus` +- `agetty` +- `auditd` +- `rsyslogd` +- `chronyd` +- `NetworkManager` +- `bash`, `sh` +- `cloud-init` +- `virtwork-*` (the chaos workload's own service) + +### Configuration + +| YAML key (`workloads.chaos-process.params.*`) | Default | Effect | +|---|---|---| +| `signal` | `SIGTERM` | Signal sent to the selected process (`SIGTERM`, `SIGKILL`, `SIGINT`, …) | +| `interval` | `30` | Seconds between kill attempts | +| `min-pid` | `1000` | Minimum PID considered eligible | + +Example: + +```yaml +workloads: + chaos-process: + enabled: true + vm_count: 1 + cpu_cores: 1 + memory: 512Mi + params: + signal: "SIGKILL" + interval: "15" + min-pid: "500" +``` + +### Example + +```bash +virtwork run --workloads chaos-process --ssh-user virtwork --ssh-key-file ~/.ssh/id_ed25519.pub + +# Watch the chaos log from inside the VM: +virtctl ssh --ssh-key ~/.ssh/id_ed25519 virtwork@virtwork-chaos-process-0 -n virtwork +journalctl -u virtwork-chaos-process.service -f +``` + +Sample output: + +``` +[2026-05-22 14:00:00] Starting chaos-process workload +[2026-05-22 14:00:00] Configuration: SIGNAL=SIGTERM INTERVAL=30s MIN_PID=1000 +[2026-05-22 14:00:00] Sending SIGTERM to PID 1437: 1437 cron /usr/sbin/cron -n +[2026-05-22 14:00:00] Successfully sent SIGTERM to PID 1437 +[2026-05-22 14:00:30] Sending SIGTERM to PID 1502: 1502 anacron /usr/sbin/anacron -d -q -s +``` + +### What to observe + +- Process restarts driven by systemd `Restart=` directives in any services you install alongside +- Alerts that fire when key processes disappear +- Recovery time from kill to restart + +--- + +## Operational Guidance + +### Running chaos alongside non-chaos workloads + +If you must run chaos workloads in the same namespace as non-chaos workloads (for example, to apply chaos to a test database), be aware that: + +- **chaos-disk** affects its own VM's data disk only — it cannot impact another VM's disk because each VM gets its own DataVolume. +- **chaos-network** affects egress from its own VM only — it does not affect traffic to or from sibling VMs. +- **chaos-process** affects processes inside its own VM only. + +Cross-VM impact happens only when the *application* under test reaches across VMs and is itself affected by the chaos (e.g., a client VM with chaos-network installed will see degraded performance talking to any server VM). + +### Cleanup + +```bash +# Drop everything chaos-related +virtwork cleanup --namespace virtwork-chaos --delete-namespace + +# Or by run ID +virtwork cleanup --run-id +``` + +Cleanup is the same as any other virtwork resource — label selectors find chaos VMs and their data volumes are reclaimed automatically when the VMs are deleted (subject to the StorageClass reclaim policy). + +### Audit visibility + +Chaos workloads are recorded in the audit database exactly like any other workload. The `workload_details` row will have `workload_type` set to `chaos-disk` / `chaos-network` / `chaos-process`, and `events` will show `vm_created`, `vm_ready`, and on cleanup `vm_deleted` rows. See [audit-schema.md](audit-schema.md). + +--- + +## For Contributors + +Source files: + +- `internal/workloads/chaos_disk.go` — `ChaosDiskWorkload` struct, `chaosDiskScript`, `chaosDiskSystemdUnit` +- `internal/workloads/chaos-network.go` — `ChaosNetworkWorkload` struct, `chaosNetworkStartScript`, `chaosNetworkStopScript`, `chaosNetworkSystemdUnit` +- `internal/workloads/chaos_process.go` — `ChaosProcessWorkload` struct, `chaosProcessScript`, `chaosProcessSystemdUnit` +- `internal/workloads/registry.go` — `DefaultRegistry()` registers all three under the names `chaos-disk`, `chaos-network`, `chaos-process` + +All three embed `BaseWorkload` and use the standard `BuildCloudConfig(opts)` helper for SSH credential injection. chaos-disk additionally uses the shared `diskSetupScript(serial, mountPoint)` helper from `internal/workloads/workload.go` for the standard wait/format/mount/fstab pattern. + +The configuration knobs listed above are sourced from `WorkloadConfig.Params` — the same plumbing the tps workload uses for `file-size`, `iterations`, `duration`. See [development.md](development.md) for the multi-VM / storage-backed / configurable-params patterns. + +## Related Docs + +- [configuration.md](configuration.md) — full reference for every configuration knob, including chaos parameters +- [audit-schema.md](audit-schema.md) — where chaos-workload events land in the audit database +- [guide/02-deploying-workloads.md](guide/02-deploying-workloads.md) — Scenario 8 walks through chaos-network end-to-end +- [development.md](development.md) — adding a new workload (including a new chaos workload) diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 0000000..19e3d1c --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,268 @@ +# Configuration Reference + +Virtwork configuration flows through four sources, merged in priority order: + +``` +CLI flags > environment variables (VIRTWORK_*) > YAML config file > built-in defaults +``` + +This document is the authoritative reference: every flag, every environment variable, every YAML key, every ConfigMap entry, and every per-workload parameter. For the *summary*, see the top-level [README](../README.md#configuration). For the implementation, see `internal/config/config.go`. + +## Quick Reference + +| Surface | When to use | +|---|---| +| **CLI flags** | One-off overrides; whatever you type wins | +| **Environment variables** (`VIRTWORK_*`) | CI/CD pipelines; ConfigMap-driven in-cluster deployments | +| **YAML config file** (`--config path.yaml`) | Reproducible runs; expressing per-workload overrides | +| **Defaults** | Sensible behavior with no arguments at all | + +--- + +## Global Persistent Flags + +These apply to both `virtwork run` and `virtwork cleanup`. + +| Flag | Env Var | YAML Key | Default | Description | +|---|---|---|---|---| +| `--namespace` | `VIRTWORK_NAMESPACE` | `namespace` | `virtwork` | Kubernetes namespace for VMs and supporting resources | +| `--kubeconfig` | `VIRTWORK_KUBECONFIG` | `kubeconfig` | _empty (in-cluster, then `~/.kube/config`)_ | Path to kubeconfig file | +| `--config` | — | — | _empty_ | Path to a YAML config file to merge in | +| `--verbose` | `VIRTWORK_VERBOSE` | `verbose` | `false` | Switch the logger from INFO to DEBUG | +| `--audit` | `VIRTWORK_AUDIT` | `audit` | `true` | Enable SQLite audit tracking | +| `--no-audit` | — | — | _unset_ | Hard override: disable audit tracking regardless of `--audit` / `VIRTWORK_AUDIT` | +| `--audit-db` | `VIRTWORK_AUDIT_DB` | `audit_db` | `virtwork.db` | Path to the audit SQLite file (use `/data/virtwork.db` in-cluster) | + +**Audit precedence note:** `--no-audit` short-circuits everything. Otherwise, the `audit` boolean follows the standard chain. In-cluster, set `VIRTWORK_AUDIT=false` in the ConfigMap to disable. + +--- + +## `virtwork run` Flags + +| Flag | Env Var | YAML Key | Default | Description | +|---|---|---|---|---| +| `--workloads` | — | _(handled via `workloads` map per-key)_ | All nine workloads, sorted: `chaos-disk, chaos-network, chaos-process, cpu, database, disk, memory, network, tps` | Comma-separated list of workloads to deploy | +| `--vm-count` | — | `workloads..vm_count` | `1` | VMs per workload (multi-VM workloads multiply by `len(Roles())`, so 1 becomes 2 for network and tps) | +| `--cpu-cores` | `VIRTWORK_CPU_CORES` | `cpu_cores` / `workloads..cpu_cores` | `2` | CPU cores per VM (per-workload override beats global) | +| `--memory` | `VIRTWORK_MEMORY` | `memory` / `workloads..memory` | `2Gi` | Memory per VM | +| `--disk-size` | `VIRTWORK_DATA_DISK_SIZE` | `data_disk_size` | `10Gi` | Data-disk size for storage-backed workloads (disk, database, chaos-disk) | +| `--container-disk-image` | `VIRTWORK_CONTAINER_DISK_IMAGE` | `container_disk_image` | `quay.io/containerdisks/fedora:41` | Boot image for the VMs; set to the golden image for faster boot | +| `--dry-run` | `VIRTWORK_DRY_RUN` | `dry_run` | `false` | Print the VM YAML and exit; do not connect to a cluster | +| `--no-wait` | — | (sets `wait_for_ready=false`) | _unset_ | Skip readiness polling after VMs are created | +| `--timeout` | `VIRTWORK_TIMEOUT` | `timeout` | `600` | Readiness timeout in seconds | +| `--ssh-user` | `VIRTWORK_SSH_USER` | `ssh_user` | `virtwork` | Username for the in-VM user account (only created when at least one SSH credential is provided) | +| `--ssh-password` | `VIRTWORK_SSH_PASSWORD` | `ssh_password` | _empty_ | Password for the SSH user (plain-text in the VM spec — prefer keys) | +| `--ssh-key` | — | `ssh_authorized_keys` (YAML list) | _empty_ | Inline SSH public key; repeatable | +| `--ssh-key-file` | — | _(handled as inline key after reading)_ | _empty_ | Path to a public key file; repeatable | +| _(env only)_ | `VIRTWORK_SSH_AUTHORIZED_KEYS` | `ssh_authorized_keys` | _empty_ | Comma-separated list of inline keys (env-var form) | + +--- + +## `virtwork cleanup` Flags + +| Flag | Env Var | YAML Key | Default | Description | +|---|---|---|---|---| +| `--delete-namespace` | — | — | `false` | Also delete the namespace itself after deleting managed resources | +| `--run-id` | — | — | _empty_ | Limit cleanup to resources labeled `virtwork/run-id=` | +| `--dry-run` | — | — | `false` | Print intent without actually deleting | + +Global flags (`--namespace`, `--kubeconfig`, `--audit`, `--no-audit`, `--audit-db`, `--verbose`) also apply. + +--- + +## Environment Variables (Alphabetical) + +| Variable | Type | Default | Source | Description | +|---|---|---|---|---| +| `VIRTWORK_AUDIT` | bool | `true` | flag-bound | Enable audit tracking | +| `VIRTWORK_AUDIT_DB` | string | `virtwork.db` | flag-bound | Audit SQLite path | +| `VIRTWORK_CONTAINER_DISK_IMAGE` | string | `quay.io/containerdisks/fedora:41` | flag-bound | VM boot image | +| `VIRTWORK_CPU_CORES` | int | `2` | flag-bound | Default per-VM CPU cores | +| `VIRTWORK_DATA_DISK_SIZE` | string | `10Gi` | flag-bound | Default data disk size | +| `VIRTWORK_DRY_RUN` | bool | `false` | flag-bound | Default for `--dry-run` | +| `VIRTWORK_KUBECONFIG` | string | _empty_ | flag-bound | Kubeconfig path | +| `VIRTWORK_MEMORY` | string | `2Gi` | flag-bound | Default per-VM memory | +| `VIRTWORK_NAMESPACE` | string | `virtwork` | flag-bound | Target namespace | +| `VIRTWORK_SSH_AUTHORIZED_KEYS` | string (csv) | _empty_ | env-only | Comma-separated SSH public keys | +| `VIRTWORK_SSH_PASSWORD` | string | _empty_ | flag-bound | SSH password | +| `VIRTWORK_SSH_USER` | string | `virtwork` | flag-bound | SSH username | +| `VIRTWORK_TIMEOUT` | int | `600` | flag-bound | Readiness timeout seconds | +| `VIRTWORK_VERBOSE` | bool | `false` | flag-bound | Verbose logging | +| `VIRTWORK_WAIT_FOR_READY` | bool | `true` | flag-bound | Inverse of `--no-wait` | +| `VIRTWORK_COMMAND` | string | _empty_ | deployment only | In-pod auto-run command: `run`, `cleanup`, or empty (sleep). Read by `entrypoint.sh`, not Viper. | +| `VIRTWORK_ARGS` | string | _empty_ | deployment only | Extra arguments when `VIRTWORK_COMMAND` is set. Read by `entrypoint.sh`, not Viper. | + +`VIRTWORK_COMMAND` and `VIRTWORK_ARGS` are container entrypoint behavior — they tell `entrypoint.sh` whether to run virtwork immediately or to sleep until invoked via `oc exec`. See [deployment.md](deployment.md). + +--- + +## YAML Config File + +Full schema with every supported key: + +```yaml +# Global defaults +namespace: virtwork-prod +container_disk_image: quay.io/opdev/virtwork-disk:latest # set to golden image to skip first-boot package installs +data_disk_size: 20Gi +cpu_cores: 2 +memory: 2Gi + +# Cluster connection (omit to use in-cluster or ~/.kube/config) +kubeconfig: /etc/virtwork/kubeconfig + +# Behavior +dry_run: false +verbose: false +wait_for_ready: true +timeout: 900 + +# SSH (optional — when omitted, no user account is created in the VM) +ssh_user: virtwork +ssh_password: "" # prefer keys +ssh_authorized_keys: + - ssh-ed25519 AAAAC3Nz... key-1 + - ssh-ed25519 AAAAC3Nz... key-2 + +# Audit +audit: true +audit_db: /data/virtwork.db + +# Per-workload overrides (everything optional; unspecified keys inherit globals) +workloads: + cpu: + enabled: true + vm_count: 2 + cpu_cores: 4 + memory: 4Gi + memory: + enabled: true + vm_count: 1 + disk: + enabled: true + database: + enabled: true + cpu_cores: 2 + memory: 4Gi + network: + enabled: true + vm_count: 1 # creates 1 server + 1 client = 2 VMs + tps: + enabled: true + vm_count: 1 # creates 1 server + 1 client = 2 VMs + params: + file-size: "50M" # default 10M + iterations: "10" # default 30 + duration: "30" # default 60 (seconds per iteration) + chaos-disk: + enabled: true + params: + mount: /mnt/data # default /mnt/data + fill-percent: "80" # default 90 + fill-sleep: "120" # default 60 + release-sleep: "60" # default 30 + chaos-network: + enabled: true + params: + latency-ms: "250" # default 100 + packet-loss-percent: "10" # default 5.0 + chaos-process: + enabled: true + params: + signal: "SIGKILL" # default SIGTERM + interval: "15" # default 30 + min-pid: "500" # default 1000 +``` + +### Per-workload `params` keys + +Each workload's `params` block accepts string-valued keys. The current parameters per workload: + +| Workload | Key | Default | Effect | +|---|---|---|---| +| **tps** | `file-size` | `10M` | Size of the HTTP test file (suffix `K`/`M`/`G`) | +| **tps** | `iterations` | `30` | Number of test iterations | +| **tps** | `duration` | `60` | Seconds per iteration | +| **chaos-disk** | `mount` | `/mnt/data` | Mountpoint of the data disk to fill | +| **chaos-disk** | `fill-percent` | `90` | Target fill percentage | +| **chaos-disk** | `fill-sleep` | `60` | Seconds held at target fill | +| **chaos-disk** | `release-sleep` | `30` | Seconds empty before refilling | +| **chaos-network** | `latency-ms` | `100` | `netem` egress delay | +| **chaos-network** | `packet-loss-percent` | `5.0` | `netem` egress drop rate | +| **chaos-process** | `signal` | `SIGTERM` | Signal sent to victims | +| **chaos-process** | `interval` | `30` | Seconds between kills | +| **chaos-process** | `min-pid` | `1000` | Minimum PID considered eligible | + +Workloads without entries above accept no per-workload `params` today (cpu, memory, disk, database, network). + +--- + +## ConfigMap for In-Cluster Deployment + +`deploy/configmap.yaml` ships the following defaults. Edit and `oc apply -k deploy/` to change behavior of an in-cluster pod. + +| ConfigMap Key | Default | Notes | +|---|---|---| +| `VIRTWORK_NAMESPACE` | `virtwork` | The pod creates VMs in this namespace | +| `VIRTWORK_CONTAINER_DISK_IMAGE` | `quay.io/containerdisks/fedora:41` | Change to the golden image to speed up boot | +| `VIRTWORK_DATA_DISK_SIZE` | `10Gi` | | +| `VIRTWORK_CPU_CORES` | `2` | | +| `VIRTWORK_MEMORY` | `2Gi` | | +| `VIRTWORK_WAIT_FOR_READY` | `true` | | +| `VIRTWORK_TIMEOUT` | `600` | | +| `VIRTWORK_DRY_RUN` | `false` | | +| `VIRTWORK_VERBOSE` | `false` | | +| `VIRTWORK_AUDIT` | `true` | | +| `VIRTWORK_AUDIT_DB` | `/data/virtwork.db` | Mounted from the `virtwork-audit-data` PVC | +| `VIRTWORK_SSH_USER` | `virtwork` | | + +`VIRTWORK_COMMAND` and `VIRTWORK_ARGS` are set directly in the Deployment `env:` (not the ConfigMap) so that updating them triggers a pod restart. See [deployment.md](deployment.md). + +`deploy/secret.yaml` provides `VIRTWORK_SSH_PASSWORD` and `VIRTWORK_SSH_AUTHORIZED_KEYS` separately so that credentials are not in the ConfigMap. + +--- + +## Priority Chain in Detail + +```mermaid +flowchart TD + CLI["CLI flag explicitly set\ne.g. --namespace virtwork-test"] + ENV["Environment variable\ne.g. VIRTWORK_NAMESPACE=virtwork-prod"] + YAML["YAML config file\nnamespace: virtwork-staging"] + DEF["Built-in default\n'virtwork'"] + + CLI -->|wins| MERGE[Effective Config] + ENV -->|2nd| MERGE + YAML -->|3rd| MERGE + DEF -->|4th| MERGE +``` + +How it works in code (`internal/config/config.go` → `LoadConfig`): + +1. `SetDefaults(v)` seeds Viper with the built-in defaults. +2. `v.SetEnvPrefix("VIRTWORK")` + `v.AutomaticEnv()` enables automatic env-var binding (with `-` ↔ `_` replacement so `--data-disk-size` ↔ `VIRTWORK_DATA_DISK_SIZE`). +3. If `--config ` is set, `v.ReadInConfig()` merges the YAML file (overrides env and defaults). +4. `bindFlagIfSet(...)` walks each known flag and copies the value into Viper *only if* the flag was explicitly set on the command line. This makes CLI flags the top of the stack without clobbering env/YAML values from defaults. +5. The `Config` struct is populated from Viper's effective view. + +Workload-specific config (`workloads..*`) is parsed as a separate map in step 5 via `v.UnmarshalKey("workloads", ...)` — it only exists in YAML; there is no env-var or flag form. + +The `--no-audit` flag is checked separately by `initAuditor` in `cmd/virtwork/main.go` and short-circuits the standard chain. + +--- + +## For Contributors + +- All config field definitions live in `internal/config/config.go` — `WorkloadConfig` struct, `Config` struct, `SetDefaults`, `BindFlags`, `LoadConfig`. +- The mapstructure tag on each field is the YAML key name (e.g., `mapstructure:"data-disk-size"` → YAML key `data_disk_size` after the `_`/`-` replacement). Match this convention when adding new fields. +- Per-workload `params` are surfaced into the workload as `WorkloadConfig.Params map[string]string`. New chaos / multi-VM knobs go here so they're uniformly addressable from YAML. +- When adding a new env var, prefer letting Viper bind it automatically rather than hand-rolling `os.Getenv` (see `VIRTWORK_SSH_AUTHORIZED_KEYS` in `resolveSSHKeys` for the one current exception, driven by the comma-split list semantics). +- Update this document, the ConfigMap default list, and the per-workload `params` table whenever you add a new knob. + +## Related Docs + +- [README.md](../README.md#configuration) — configuration summary in the main project README +- [chaos-workloads.md](chaos-workloads.md) — narrative description of how chaos params shape workload behavior +- [deployment.md](deployment.md) — ConfigMap + Secret + Deployment env semantics +- [development.md](development.md) — adding new workloads (including new params) +- [audit-schema.md](audit-schema.md) — audit-related configuration consequences diff --git a/docs/deployment.md b/docs/deployment.md new file mode 100644 index 0000000..07670b6 --- /dev/null +++ b/docs/deployment.md @@ -0,0 +1,226 @@ +# OpenShift Deployment + +Virtwork can run as a pod on the cluster it operates against, using the Kustomize manifests in [`deploy/`](../deploy). This is the right deployment model when you want virtwork to be reachable from `oc exec`, scheduled by CI, or driven by a GitOps workflow. + +For the smaller "run from my laptop with my kubeconfig" mode, just build and execute the binary — no manifests are needed. + +## What `oc apply -k deploy/` Creates + +```mermaid +flowchart TB + NS["Namespace
virtwork"] + + subgraph Identity + SA["ServiceAccount
virtwork"] + CR["ClusterRole
virtwork
(VMs, VMIs, DVs,
Services, Secrets, Namespaces)"] + CRB["ClusterRoleBinding
virtwork"] + end + + subgraph Config + CM["ConfigMap
virtwork-config
VIRTWORK_* defaults"] + SEC["Secret
virtwork-secrets
VIRTWORK_SSH_PASSWORD
VIRTWORK_SSH_AUTHORIZED_KEYS"] + end + + subgraph Persistence + PVC["PersistentVolumeClaim
virtwork-audit-data
RWO, 1Gi"] + end + + subgraph Workload + DEP["Deployment
virtwork
replicas: 1, Recreate strategy"] + POD["Pod
(entrypoint.sh)"] + end + + SA --> CRB + CR --> CRB + CRB --> SA + + NS --> SA + NS --> CM + NS --> SEC + NS --> PVC + NS --> DEP + DEP --> POD + SA --> POD + CM --> POD + SEC --> POD + PVC --> POD +``` + +| Resource | File | Purpose | +|---|---|---| +| Namespace `virtwork` | `namespace.yaml` | Isolation boundary for virtwork-managed resources | +| ServiceAccount `virtwork` | `serviceaccount.yaml` | Identity the pod runs as | +| ClusterRole + ClusterRoleBinding `virtwork` | `rbac.yaml` | Permissions to manage VMs, VMIs, DataVolumes, Services, Secrets, Namespaces — cluster-scoped because Namespaces themselves are cluster-scoped | +| ConfigMap `virtwork-config` | `configmap.yaml` | Default `VIRTWORK_*` env vars | +| Secret `virtwork-secrets` | `secret.yaml` | SSH credentials (kept out of the ConfigMap) | +| PersistentVolumeClaim `virtwork-audit-data` | `pvc.yaml` | 1Gi RWO claim mounted at `/data` for the audit SQLite file | +| Deployment `virtwork` | `deployment.yaml` | The pod itself | + +The Kustomization (`deploy/kustomization.yaml`) applies `app.kubernetes.io/name: virtwork` and `app.kubernetes.io/managed-by: virtwork` labels to every resource. + +## Deploy + +```bash +oc apply -k deploy/ + +# Verify +oc get all,configmap,secret,pvc -n virtwork +``` + +The Deployment uses `strategy.type: Recreate` because the audit PVC is `ReadWriteOnce` — a rolling update would attempt to schedule two pods simultaneously and the new pod would fail to bind the volume. + +## Container Behavior: `VIRTWORK_COMMAND` and `VIRTWORK_ARGS` + +The container's behavior is governed by two environment variables defined directly in the Deployment (not the ConfigMap, so that changing them rolls the pod): + +| Env | Effect | +|---|---| +| `VIRTWORK_COMMAND` empty (default) | Pod runs `sleep infinity`. Use `oc exec` to run virtwork commands by hand. | +| `VIRTWORK_COMMAND=run` | Pod auto-executes `virtwork run $VIRTWORK_ARGS` on start. | +| `VIRTWORK_COMMAND=cleanup` | Pod auto-executes `virtwork cleanup $VIRTWORK_ARGS` on start. | + +The entrypoint script ([`entrypoint.sh`](../entrypoint.sh)) validates `VIRTWORK_COMMAND` (only `run` and `cleanup` are accepted) and `exec`s into virtwork. Pod stdout/stderr ends up in the container log. + +**Important:** `VIRTWORK_ARGS` is intentionally unquoted to allow space-delimited tokens (`"--workloads cpu,memory --vm-count 2"`). As a consequence, **no argument value can contain a space.** + +### Interactive use + +```bash +oc exec -it deploy/virtwork -- virtwork run --dry-run +oc exec -it deploy/virtwork -- virtwork run --workloads cpu,memory +oc exec -it deploy/virtwork -- virtwork cleanup +oc exec -it deploy/virtwork -- sqlite3 /data/virtwork.db "SELECT run_id, command, status, started_at FROM audit_log ORDER BY id DESC LIMIT 5" +``` + +### Auto-execute on pod start + +```bash +oc set env deploy/virtwork VIRTWORK_COMMAND=run VIRTWORK_ARGS="--workloads cpu,memory --vm-count 2" +``` + +The deployment rolls the pod, which executes virtwork and exits. The audit row persists on the PVC. For repeatable scheduled runs, drive `VIRTWORK_COMMAND` / `VIRTWORK_ARGS` from a CronJob or GitOps tool rather than the Deployment directly. + +## Image Pinning + +The Deployment ships pinned to a semantic version (`quay.io/opdev/virtwork:v0.0.1`) for reproducibility. Bump the tag when upgrading. + +```yaml +containers: + - name: virtwork + image: quay.io/opdev/virtwork:v0.0.1 # update on upgrade + imagePullPolicy: IfNotPresent +``` + +For development you may pin to `:latest` instead, but be aware that `IfNotPresent` plus `:latest` can pin a node to an older copy of the image until it's pulled again. Either bump the tag, or set `imagePullPolicy: Always` for floating tags. + +The container image is built from [`Dockerfile`](../Dockerfile) (multi-stage: `golang:1.25-bookworm` builder for CGO + `ubi9/ubi-minimal` runtime with `sqlite-libs`). A CI variant lives at [`Dockerfile.ci`](../Dockerfile.ci). + +```bash +podman build -t quay.io/opdev/virtwork:vX.Y.Z . +podman push quay.io/opdev/virtwork:vX.Y.Z +``` + +Then update `deploy/deployment.yaml` and `oc apply -k deploy/`. + +### Using the Golden Container Disk Image + +The golden image at `quay.io/opdev/virtwork-disk:latest` pre-installs every workload's tools so VMs boot ready-to-run. To make it the default for an in-cluster deployment, set the ConfigMap key: + +```bash +oc set data configmap/virtwork-config -n virtwork VIRTWORK_CONTAINER_DISK_IMAGE=quay.io/opdev/virtwork-disk:latest +``` + +The Deployment uses `envFrom` to pull all ConfigMap keys into the pod, so the change takes effect on the next `virtwork run`. No pod restart needed. See [../build/golden-image/README.md](../build/golden-image/README.md) for build instructions. + +## RBAC Scope + +The ClusterRole `virtwork` grants: + +| API Group | Resources | Verbs | Why | +|---|---|---|---| +| `""` (core) | `namespaces` | `create`, `get`, `delete` | `EnsureNamespace`; `cleanup --delete-namespace` | +| `kubevirt.io` | `virtualmachines` | `create`, `delete`, `get`, `list` | Full VM lifecycle | +| `kubevirt.io` | `virtualmachineinstances` | `get`, `list` | Readiness polling (`GetVMIPhase`) | +| `cdi.kubevirt.io` | `datavolumes` | `create`, `delete`, `get`, `list` | DataVolumes are created via embedded `DataVolumeTemplates` in VMs | +| `""` (core) | `services` | `create`, `delete`, `get`, `list` | Multi-VM workloads (network, tps) need ClusterIP Services | +| `""` (core) | `secrets` | `create`, `delete`, `get`, `list` | Cloud-init userdata Secrets (one per VM) | + +This is a ClusterRole — not a Role — because `namespaces` are cluster-scoped. Permissions are intentionally broad to support cross-namespace test setups; tighten the binding to a single namespace if your environment requires it. + +## Sizing + +The Deployment's defaults (`requests: 64Mi / 100m`, `limits: 256Mi / 500m`) are sized for the orchestrator process itself, not the VMs it creates. Bump the limits if: + +- You deploy all nine workloads simultaneously (11 VMs of plumbing in one pod) — concurrent VM-create goroutines and audit writes consume modestly more memory. +- You consume the audit DB with long-running queries from within the pod. + +The VMs themselves are sized via `--cpu-cores`, `--memory`, and per-workload YAML overrides; their resource requests come from the VirtualMachine spec, not the virtwork pod. + +## DataVolume Names + +When you list DataVolumes inside the namespace, you'll see entries like: + +``` +$ oc get dv -n virtwork +NAME PHASE AGE +virtwork-disk-data-virtwork-disk-0 Succeeded 3m +virtwork-database-data-virtwork-database-0 Succeeded 3m +virtwork-chaos-disk-data-virtwork-chaos-disk-0 Succeeded 3m +``` + +The suffix (`-virtwork-disk-0`, `-virtwork-chaos-disk-0`) comes from `namespaceDataVolumes` in `cmd/virtwork/main.go`. Each `VMCount > 1` deployment gets its own uniquely-named DataVolume so multiple VMs of the same workload type don't collide on a single namespace-scoped DV name. The base name (`virtwork-disk-data`, etc.) is what the workload declares in its `DataVolumeTemplates()` method; the orchestrator appends the VM name. + +## Persisting the Audit Database + +The audit DB lives at `/data/virtwork.db` in-pod, mounted from the `virtwork-audit-data` PVC. The default PVC is 1Gi RWO — plenty for tens of thousands of executions. + +The PVC is intentionally separate from the Deployment so that `oc apply -k deploy/` upgrades preserve audit history. + +To inspect from outside the cluster: + +```bash +# Copy the DB out +oc cp virtwork/$(oc get pod -n virtwork -l app.kubernetes.io/name=virtwork -o name | head -1 | cut -d/ -f2):/data/virtwork.db ./virtwork.db +sqlite3 virtwork.db "SELECT run_id, command, status, started_at FROM audit_log ORDER BY id DESC LIMIT 10" +``` + +Or query in-place: + +```bash +oc exec -n virtwork deploy/virtwork -- sqlite3 /data/virtwork.db "SELECT run_id, command, status, started_at FROM audit_log ORDER BY id DESC LIMIT 10" +``` + +See [audit-schema.md](audit-schema.md) for the full schema and more queries. + +## Security Context + +The pod runs: + +- `runAsNonRoot: true` +- `seccompProfile.type: RuntimeDefault` +- `allowPrivilegeEscalation: false` +- `capabilities.drop: [ALL]` + +These are pod- and container-level defaults that the security review process expects. Override only if you have a specific need. + +The Secret (`virtwork-secrets`) carries the SSH password and authorized-keys list. It is mounted via `envFrom` so the values appear as env vars inside the pod. **Do not store SSH passwords as plain text in production** — prefer authorized keys via `VIRTWORK_SSH_AUTHORIZED_KEYS`, and even then treat the secret as sensitive in your cluster RBAC. + +## Uninstalling + +```bash +# Just the pod and identity, keeping the audit history +oc delete deployment virtwork -n virtwork + +# Or the whole thing including the namespace and PVC +oc delete -k deploy/ +``` + +`oc delete -k deploy/` removes everything in the Kustomize set, including the PVC. If you want to keep the audit history across uninstalls, omit `pvc.yaml` from the deletion or delete resources individually. + +## Related Docs + +- [README.md](../README.md#openshift-deployment) — the high-level deployment summary +- [configuration.md](configuration.md) — every ConfigMap key with its meaning and default +- [audit-schema.md](audit-schema.md) — what's in the PVC-backed SQLite file +- [chaos-workloads.md](chaos-workloads.md) — running chaos workloads from an in-cluster pod +- [../build/golden-image/README.md](../build/golden-image/README.md) — building and using the optional golden container disk image diff --git a/docs/development.md b/docs/development.md index 6b5a13d..4b74a98 100644 --- a/docs/development.md +++ b/docs/development.md @@ -421,20 +421,28 @@ internal/ # Application packages (not importable externally) config/ # Config struct, Viper priority chain cluster/ # controller-runtime client init + scheme cloudinit/ # Cloud-config YAML builder + logging/ # Structured slog logger (verbose -> DEBUG) vm/ # VM spec construction + typed CRUD + retry resources/ # Namespace + Service + Secret helpers wait/ # VMI readiness polling (errgroup) cleanup/ # Label-based teardown (VMs, Services, Secrets) audit/ # SQLite audit tracking (Auditor interface, schema, records) - workloads/ # Workload interface, 5 implementations, registry + workloads/ # Workload + MultiVMWorkload interfaces, nine implementations, registry testutil/ # Shared test helpers for integration and E2E tests tests/ # Tests requiring external infrastructure e2e/ # E2E acceptance tests (//go:build e2e) +build/ + golden-image/ # Optional Fedora container disk with pre-installed tools +deploy/ # Kustomize manifests for OpenShift deployment docs/ # Documentation + README.md # Documentation index architecture.md # Layered architecture and mermaid diagrams - implementation-plan.md # Phased build plan development.md # This file - engineering-journals/ # Per-phase development journals + configuration.md # Complete configuration reference + deployment.md # OpenShift deployment deep-dive + audit-schema.md # SQLite audit schema reference + chaos-workloads.md # Chaos engineering workload guide + guide/ # Hands-on guides (overview, deploying, adding workloads) ``` ## Architecture Layers @@ -444,7 +452,7 @@ The codebase follows a strict layered architecture where each layer depends only | Layer | Packages | Goroutines | Purpose | |-------|----------|------------|---------| | 0 | `constants` | No | Pure values — API coordinates, labels, defaults | -| 1 | `config`, `cloudinit`, `cluster` | No | Configuration, cloud-init YAML, K8s client init | +| 1 | `config`, `cloudinit`, `cluster`, `logging` | No | Configuration, cloud-init YAML, K8s client init, structured logging | | 2 | `vm`, `resources`, `wait` | Yes | K8s CRUD operations with retry, readiness polling | | 3 | `workloads` | No | Pure data producers — cloud-init specs, resource structs | | 4 | `cmd/virtwork`, `cleanup`, `audit` | Yes | Orchestration, teardown, and audit tracking | @@ -531,17 +539,21 @@ func (w *MyWorkload) UserdataForRole(role string, namespace string) (string, err ### 3. Register the Workload -Add the constructor to `internal/workloads/registry.go`: +Add the constructor to `internal/workloads/registry.go`. `DefaultRegistry()` currently has nine entries; add yours alongside: ```go func DefaultRegistry() Registry { return Registry{ - "cpu": func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewCPUWorkload(cfg, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) }, - "memory": func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewMemoryWorkload(cfg, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) }, - "database": func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewDatabaseWorkload(cfg, opts.DataDiskSize, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) }, - "network": func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewNetworkWorkload(cfg, opts.Namespace, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) }, - "disk": func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewDiskWorkload(cfg, opts.DataDiskSize, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) }, - "my-workload": func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewMyWorkload(cfg, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) }, + "cpu": func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewCPUWorkload(cfg, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) }, + "memory": func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewMemoryWorkload(cfg, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) }, + "disk": func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewDiskWorkload(cfg, opts.DataDiskSize, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) }, + "database": func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewDatabaseWorkload(cfg, opts.DataDiskSize, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) }, + "network": func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewNetworkWorkload(cfg, opts.Namespace, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) }, + "tps": func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewTPSWorkload(cfg, opts.Namespace, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) }, + "chaos-disk": func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewChaosDiskWorkload(cfg, opts.DataDiskSize, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) }, + "chaos-network": func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewChaosNetworkWorkload(cfg, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) }, + "chaos-process": func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewChaosProcessWorkload(cfg, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) }, + "my-workload": func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewMyWorkload(cfg, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) }, } } ``` @@ -551,6 +563,55 @@ func DefaultRegistry() Registry { - Orchestration BDD tests will fail (total VM count assertions) - Update both before considering the feature complete +### Going Further: Multi-VM Workloads + +If your workload needs more than one role of VM (e.g., a server and a client), implement the `MultiVMWorkload` interface in addition to `Workload`: + +```go +// In internal/workloads/workload.go +type MultiVMWorkload interface { + Workload + Roles() []string + UserdataForRole(role string, namespace string) (string, error) +} +``` + +Pattern: + +1. Embed `BaseWorkload` and store any per-workload state (e.g., a `Namespace` field for in-cluster DNS). +2. Override `VMCount()` to return `count * len(Roles())` so the orchestrator creates the right number of VMs per role. +3. Override `Roles()` to return the role names (e.g., `[]string{"server", "client"}`). +4. Override `UserdataForRole(role, namespace)` to return role-specific cloud-init. The default `CloudInitUserdata()` typically delegates to `UserdataForRole("server", namespace)`. +5. Set `RequiresService()` to `true` and provide a `ServiceSpec()` selecting server VMs by the `virtwork/role: server` label that the orchestrator applies automatically. +6. Clients reach servers via the in-cluster DNS name `..svc.cluster.local` — never poll for pod IPs. + +The canonical references are `internal/workloads/network.go` (simplest — one port, iperf3) and `internal/workloads/tps.go` (multi-port Service with configurable `Params` for `file-size`, `iterations`, `duration`). + +### Going Further: Storage-Backed Workloads + +If your workload needs persistent storage inside the VM: + +1. Override `DataVolumeTemplates()` to return a CDI `DataVolume` for each volume needed. Use `vm.BuildDataVolumeTemplate(name, size)` from `internal/vm`. +2. Override `ExtraVolumes()` and `ExtraDisks()` to wire the DataVolume into the VM. **Always set the `Serial` field on the `Disk`** — the in-VM script discovers the device through `/dev/disk/by-id/virtio-`, which is deterministic across reboots (unlike `/dev/vdX`, which is not). +3. In your cloud-init userdata, write the shared `diskSetupScript(serial, mountPoint)` helper (from `internal/workloads/workload.go`) as the first script. It waits for the symlink, formats with XFS if empty, mounts, and writes `/etc/fstab` for persistence across reboots. +4. The orchestrator's `namespaceDataVolumes` helper automatically suffixes DV template names with the VM name to avoid collisions across multiple VMs of the same workload. Your template name should be the un-suffixed base (e.g., `virtwork-chaos-disk-data`). + +Reference workloads: `disk.go` (single fio mount), `database.go` (PostgreSQL data dir), `chaos_disk.go` (fill/release loop). All three follow the same pattern. + +### Going Further: Structured Logging + +The `internal/logging` package provides a shared `*slog.Logger` returned by `NewLogger(w io.Writer, verbose bool)`. Use it instead of `fmt.Fprintf` or `log.Printf` in any code path under `cmd/` or in packages that perform I/O (`internal/wait` is the current example): + +```go +logger := logging.NewLogger(cmd.OutOrStdout(), verbose) +logger.Info("vm created", + slog.String("vm_name", name), + slog.String("namespace", ns), + slog.String("workload", component)) +``` + +The output is JSON. `--verbose` flips the level from `INFO` to `DEBUG`. Workload constructors and the pure data layer (`internal/workloads`, `internal/cloudinit`) should not log — they remain pure data producers. + ### 4. Write Tests Create `internal/workloads/my_workload_test.go` using Ginkgo: @@ -734,6 +795,19 @@ This project uses [Conventional Commits](https://www.conventionalcommits.org/): | `refactor:` | Code restructuring without behavior change | | `chore:` | Build, tooling, or maintenance | +Every commit must include a DCO `Signed-off-by` trailer matching the author's identity; a `commit-msg` hook enforces this. + +--- + +## Related Documentation + +- [architecture.md](architecture.md) — Layered architecture, mermaid diagrams, key design decisions +- [configuration.md](configuration.md) — Complete config reference (flags, env vars, YAML keys, ConfigMap) +- [audit-schema.md](audit-schema.md) — SQLite schema for the audit database +- [chaos-workloads.md](chaos-workloads.md) — Operator guide for chaos-disk, chaos-network, chaos-process +- [deployment.md](deployment.md) — OpenShift deployment via Kustomize +- [guide/03-adding-a-workload.md](guide/03-adding-a-workload.md) — TDD walkthrough that builds a new workload from scratch + ## Idempotency and Safety - `apierrors.IsAlreadyExists()` responses are treated as success (resource already exists) diff --git a/docs/guide/01-overview.md b/docs/guide/01-overview.md index fcdfdec..e433156 100644 --- a/docs/guide/01-overview.md +++ b/docs/guide/01-overview.md @@ -208,17 +208,23 @@ func (b *BaseWorkload) RequiresService() bool { return false } // ... etc ``` -Concrete workloads embed `BaseWorkload` and only override what they need. This creates a natural complexity spectrum: +Concrete workloads embed `BaseWorkload` and only override what they need. This creates a natural complexity spectrum across all nine built-in workloads: | Workload | Overrides Beyond Name + CloudInit | Complexity | |----------|-----------------------------------|------------| | **CPU** | Nothing | Simplest | | **Memory** | Nothing | Simplest | -| **Database** | `DataVolumeTemplates`, `ExtraDisks`, `ExtraVolumes` | Medium | -| **Disk** | `DataVolumeTemplates`, `ExtraDisks`, `ExtraVolumes` | Medium | -| **Network** | `VMCount`, `RequiresService`, `ServiceSpec`, plus `MultiVMWorkload` interface | Most complex | +| **Chaos-process** | Nothing (systemd unit defaults baked in) | Simplest | +| **Chaos-network** | Nothing in Go (`Latency`, `PacketLoss` baked into struct) | Simple | +| **Disk** | `DataVolumeTemplates`, `ExtraDisks`, `ExtraVolumes` (virtio Serial + `diskSetupScript`) | Medium | +| **Database** | `DataVolumeTemplates`, `ExtraDisks`, `ExtraVolumes` (virtio Serial + `diskSetupScript`) | Medium | +| **Chaos-disk** | `DataVolumeTemplates`, `ExtraDisks`, `ExtraVolumes` (virtio Serial + `diskSetupScript`) | Medium | +| **Network** | `VMCount`, `Roles`, `UserdataForRole`, `RequiresService`, `ServiceSpec` (implements `MultiVMWorkload`) | Most complex | +| **TPS** | `VMCount`, `Roles`, `UserdataForRole`, `RequiresService`, `ServiceSpec`, `Params` (implements `MultiVMWorkload`) | Most complex | -The network workload is the most involved — it creates two VMs per configured count (a server and a client), needs a Kubernetes Service for DNS routing between them, and generates different cloud-init YAML for each role. +The multi-VM workloads (network, tps) are the most involved — they create two VMs per configured count (a server and a client), need a Kubernetes Service for DNS routing between them, and generate different cloud-init YAML for each role. + +`MultiVMWorkload` extends `Workload` with `Roles()` and `UserdataForRole(role, namespace)`. The orchestrator type-asserts to this interface and dispatches per role; see [development.md](../development.md) for the implementation pattern. ```mermaid flowchart LR @@ -263,15 +269,18 @@ If you want to dig deeper into how each component works: | I want to understand... | Look at... | |------------------------|------------| -| How workloads define themselves | `internal/workloads/` — interface in `workload.go`, implementations in `cpu.go`, `memory.go`, etc. | +| How workloads define themselves | `internal/workloads/` — `Workload` and `MultiVMWorkload` interfaces in `workload.go`; implementations in `cpu.go`, `memory.go`, `disk.go`, `database.go`, `network.go`, `tps.go`, `chaos_disk.go`, `chaos-network.go`, `chaos_process.go` | +| The `diskSetupScript` helper for storage-backed workloads | `internal/workloads/workload.go` — generates the wait/format/mount/fstab script for `/dev/disk/by-id/virtio-` | | How VMs are built from workload data | `internal/vm/vm.go` — `BuildVMSpec()` and `CreateVM()` | -| The CLI orchestration flow | `cmd/virtwork/main.go` — `runE()` and `cleanupE()` | +| The CLI orchestration flow | `cmd/virtwork/main.go` — `runE()` and `cleanupE()`, including `namespaceDataVolumes` for per-VM DV naming | | Configuration loading | `internal/config/config.go` — `LoadConfig()` with Viper | | Cloud-init YAML generation | `internal/cloudinit/cloudinit.go` — `BuildCloudConfig()` | | Resource helpers (namespace, service, secret) | `internal/resources/resources.go` | | VM readiness polling | `internal/wait/wait.go` — concurrent VMI phase polling | | Cleanup by label selector | `internal/cleanup/cleanup.go` — `CleanupAll()` | -| Audit logging | `internal/audit/` — `Auditor` interface, SQLite schema, records | +| Audit logging | `internal/audit/` — `Auditor` interface, SQLite schema, records (see [audit-schema.md](../audit-schema.md) for the full schema) | +| Structured logging | `internal/logging/logging.go` — `NewLogger(out, verbose)` returning a `*slog.Logger` | | Constants and defaults | `internal/constants/constants.go` | +| Golden container disk image | `build/golden-image/` — Containerfile and build script for the optional pre-tooled image | For the full layered architecture with dependency diagrams, see [docs/architecture.md](../architecture.md). diff --git a/docs/guide/02-deploying-workloads.md b/docs/guide/02-deploying-workloads.md index 07566cb..c11eb7e 100644 --- a/docs/guide/02-deploying-workloads.md +++ b/docs/guide/02-deploying-workloads.md @@ -200,20 +200,25 @@ This creates 3 VMs, 3 cloud-init secrets, and 0 services: virtwork run ``` -With no `--workloads` flag, all workload types deploy. This creates **8 VMs** total — the network and tps workloads each create server/client pairs: +With no `--workloads` flag, all nine workload types deploy. This creates **11 VMs** total — seven single-VM workloads plus the network and tps workloads, which each create a server/client pair: | VM Name | Workload | Role | |---------|----------|------| | `virtwork-cpu-0` | CPU | — | -| `virtwork-database-0` | Database | — | -| `virtwork-disk-0` | Disk | — | | `virtwork-memory-0` | Memory | — | +| `virtwork-disk-0` | Disk | — | +| `virtwork-database-0` | Database | — | | `virtwork-network-server-0` | Network | server | | `virtwork-network-client-0` | Network | client | | `virtwork-tps-server-0` | TPS | server | | `virtwork-tps-client-0` | TPS | client | +| `virtwork-chaos-disk-0` | Chaos-disk | — | +| `virtwork-chaos-network-0` | Chaos-network | — | +| `virtwork-chaos-process-0` | Chaos-process | — | + +The network workload creates a `virtwork-iperf3-server` ClusterIP Service on port 5201. The tps workload creates a `virtwork-tps-server` ClusterIP Service on ports 12865 (netperf control), 12866 (netperf data), and 8080 (HTTP file transfer). The chaos workloads do not create Services — their behavior is confined to the VM they run in. -The network workload creates a `virtwork-iperf3-server` ClusterIP Service on port 5201. The tps workload creates a `virtwork-tps-server` ClusterIP Service on ports 12865 (netperf control), 12866 (netperf data), and 8080 (HTTP file transfer). +> ⚠️ The chaos workloads inject destructive behavior inside their VMs (disk fill, network latency/loss, random process kills). They are safe to run in an isolated namespace but should never share a namespace with workloads you care about. See [chaos-workloads.md](../chaos-workloads.md) for operational guidance. ### Scaling up @@ -435,6 +440,71 @@ Set the `VIRTWORK_COMMAND` and `VIRTWORK_ARGS` environment variables in the Depl oc set env deploy/virtwork VIRTWORK_COMMAND=run VIRTWORK_ARGS="--workloads cpu,memory --vm-count 2" ``` +See [deployment.md](../deployment.md) for a manifest-by-manifest deep dive on the Kustomize layout. + +--- + +## Scenario 8: Inject Network Chaos + +The chaos-network workload uses `tc` and `netem` to add latency and packet loss to the VM's egress interface — useful for testing how an application or monitoring stack reacts to a degraded network. + +```bash +virtwork run --workloads chaos-network --ssh-user virtwork --ssh-key-file ~/.ssh/id_ed25519.pub +``` + +After the VM is ready, SSH in and confirm the qdisc is applied: + +```bash +virtctl ssh --ssh-key ~/.ssh/id_ed25519 virtwork@virtwork-chaos-network-0 -n virtwork + +# Inside the VM: +tc qdisc show dev eth0 +``` + +You should see a `netem` qdisc with `delay 100ms` and `loss 5%` on the default-route interface. Verify with a ping to a known host: + +```bash +ping -c 5 1.1.1.1 +``` + +Expect ~100ms RTT and occasional packet loss in the output. The qdisc is managed by the `virtwork-chaos-network.service` systemd unit, so it will be reapplied on reboot. + +For chaos-disk and chaos-process, plus the full risk model, see [chaos-workloads.md](../chaos-workloads.md). + +--- + +## Scenario 9: Transaction Performance with tps + +The tps workload measures transactions per second using two complementary tests on a server/client pair: `netperf TCP_RR` (small-packet round-trips at the transport layer) and an HTTP file-transfer loop (larger transfers at the application layer). + +```bash +virtwork run --workloads tps --ssh-user virtwork --ssh-key-file ~/.ssh/id_ed25519.pub +``` + +Two VMs are created — `virtwork-tps-server-0` (runs `netserver` and a Python HTTP server) and `virtwork-tps-client-0` (loops through netperf and curl tests against the server's Service DNS name) — plus the `virtwork-tps-server` ClusterIP Service on ports 12865, 12866, and 8080. + +Watch the client's output: + +```bash +virtctl ssh --ssh-key ~/.ssh/id_ed25519 virtwork@virtwork-tps-client-0 -n virtwork + +# Inside the VM: +journalctl -u virtwork-tps.service -f +``` + +You will see alternating TCP_RR iteration blocks (transactions/sec) and HTTP iteration blocks (transfers/sec, MB/s). + +The defaults are 30 iterations × 60 seconds each, 10MB transfer file. To customize, set `workloads.tps.params` in a YAML config: + +```yaml +workloads: + tps: + params: + file-size: "50M" + iterations: "10" + duration: "30" +``` + --- ## Troubleshooting @@ -446,4 +516,5 @@ oc set env deploy/virtwork VIRTWORK_COMMAND=run VIRTWORK_ARGS="--workloads cpu,m | VMs stuck in `Provisioning` | CDI not installed or no default StorageClass | Install the OpenShift Virtualization operator and ensure a default StorageClass exists | | Timeout waiting for readiness | Slow image pull or boot | Increase `--timeout` (default is 600s), or use `--no-wait` and check manually | | Cloud-init failed inside VM | Package install failure or script error | SSH in and check `/var/log/cloud-init-output.log` | -| `unknown workload` error | Typo in `--workloads` | Available workloads: `cpu`, `database`, `disk`, `memory`, `network`, `tps` | +| chaos-network: `tc` does nothing inside the VM | `sch_netem` kernel module not loaded | Confirm the VM image has `iproute-tc` and `kernel-modules-extra` available; check `journalctl -u virtwork-chaos-network.service` | +| `unknown workload` error | Typo in `--workloads` | Available workloads: `cpu`, `memory`, `disk`, `database`, `network`, `tps`, `chaos-disk`, `chaos-network`, `chaos-process` | diff --git a/docs/guide/03-adding-a-workload.md b/docs/guide/03-adding-a-workload.md index 4bf853e..e7b8339 100644 --- a/docs/guide/03-adding-a-workload.md +++ b/docs/guide/03-adding-a-workload.md @@ -466,23 +466,30 @@ go run ./cmd/virtwork cleanup ## Going Further -### Adding a Data Disk +### Adding a Data Disk (Storage-Backed Workloads) -If your workload needs persistent storage (for example, a workload that writes benchmark results to disk), override three methods. Look at `internal/workloads/disk.go` for the complete pattern: +If your workload needs persistent storage (for example, a workload that writes benchmark results to disk), override three methods. Look at `internal/workloads/disk.go`, `database.go`, or `chaos_disk.go` for the complete pattern. The key things to get right: ```go -// DataVolumeTemplates returns a CDI DataVolumeTemplateSpec. +// DataVolumeTemplates returns a CDI DataVolumeTemplateSpec. The orchestrator +// suffixes the template name with the VM name (namespaceDataVolumes in +// cmd/virtwork/main.go) to avoid collisions when --vm-count > 1, so use the +// un-suffixed base name here. func (w *MyWorkload) DataVolumeTemplates() []kubevirtv1.DataVolumeTemplateSpec { return []kubevirtv1.DataVolumeTemplateSpec{ vm.BuildDataVolumeTemplate("my-data", w.DataDiskSize), } } -// ExtraDisks adds the disk definition to the VM spec. +// ExtraDisks adds the disk definition to the VM spec. ALWAYS set the Serial +// field — the in-VM script discovers the device via +// /dev/disk/by-id/virtio-, which is stable across reboots and +// migrations (unlike /dev/vdX, which is not). func (w *MyWorkload) ExtraDisks() []kubevirtv1.Disk { return []kubevirtv1.Disk{ { - Name: "datadisk", + Name: "datadisk", + Serial: "virtwork-mydata", DiskDevice: kubevirtv1.DiskDevice{ Disk: &kubevirtv1.DiskTarget{Bus: "virtio"}, }, @@ -490,7 +497,8 @@ func (w *MyWorkload) ExtraDisks() []kubevirtv1.Disk { } } -// ExtraVolumes links the disk to the DataVolume. +// ExtraVolumes links the disk to the DataVolume. The Name must match +// ExtraDisks; the DataVolume.Name must match the template name above. func (w *MyWorkload) ExtraVolumes() []kubevirtv1.Volume { return []kubevirtv1.Volume{ { @@ -503,27 +511,46 @@ func (w *MyWorkload) ExtraVolumes() []kubevirtv1.Volume { } ``` -All three methods must return matching `Name` values — the disk, volume, and DataVolume are linked by name. +In your cloud-init, write the shared `diskSetupScript(serial, mountPoint)` helper as the first script and run it from `runcmd` before the workload service starts. It waits for the `/dev/disk/by-id/virtio-` symlink, formats with XFS if empty, mounts at `mountPoint`, and writes `/etc/fstab` so the mount survives reboots. + +```go +return w.BuildCloudConfig(CloudConfigOpts{ + WriteFiles: []WriteFile{ + { + Path: "/usr/local/bin/virtwork-disk-setup.sh", + Content: diskSetupScript("virtwork-mydata", "/mnt/data"), + Permissions: "0755", + }, + // ... workload service unit and script ... + }, + RunCmd: [][]string{ + {"/usr/local/bin/virtwork-disk-setup.sh"}, + {"systemctl", "daemon-reload"}, + {"systemctl", "enable", "--now", "virtwork-my-workload.service"}, + }, +}) +``` ### Making It Multi-VM -If you wanted nginx on one VM and `ab` on another (for more realistic cross-network metrics), you would implement the `MultiVMWorkload` interface. See `internal/workloads/network.go` for the complete pattern: +If your workload needs more than one role of VM (a server and one or more clients, for example), implement the `MultiVMWorkload` interface. The two canonical references are `internal/workloads/network.go` (simplest — one Service port, iperf3) and `internal/workloads/tps.go` (multi-port Service with configurable `Params` for `file-size`, `iterations`, `duration`). -1. Add a `Namespace` field to your struct (the client needs the server's DNS name) -2. Implement `UserdataForRole(role, namespace) (string, error)` — return different cloud-init YAML for `"server"` vs `"client"` -3. Override `VMCount()` to return `count * 2` -4. Override `RequiresService()` to return `true` -5. Implement `ServiceSpec()` to create a ClusterIP Service targeting the server VM by label selector +1. Add a `Namespace` field to your struct — the client needs it to build the server's in-cluster DNS name. +2. Implement `Roles() []string` (e.g., `[]string{"server", "client"}`). +3. Implement `UserdataForRole(role, namespace) (string, error)` — return different cloud-init YAML per role. The orchestrator dispatches per role; the client constructs `..svc.cluster.local` and never polls for pod IPs. +4. Override `VMCount()` to return `count * len(Roles())` so the orchestrator creates the right number of VMs. +5. Override `RequiresService()` to return `true`. +6. Implement `ServiceSpec()` to create a ClusterIP Service. Its selector should match `virtwork/role: server` (and ideally `app.kubernetes.io/component: `) — the orchestrator applies the `virtwork/role` label to each VM automatically. -The orchestrator detects `MultiVMWorkload` via type assertion and calls `UserdataForRole()` for each VM instead of `CloudInitUserdata()`. +The orchestrator detects `MultiVMWorkload` via type assertion and calls `UserdataForRole()` per role/instance instead of `CloudInitUserdata()`. ### Workload Complexity Spectrum ```mermaid flowchart LR - A["Simple
CPU, Memory, HTTP
Name + CloudInit only"] - B["With Storage
Database, Disk
+ DataVolumeTemplates
+ ExtraDisks/Volumes"] - C["Multi-VM
Network
+ MultiVMWorkload
+ Service + VMCount"] + A["Simple
CPU, Memory, Chaos-process
Name + CloudInit only"] + B["With Storage
Disk, Database, Chaos-disk
+ DataVolumeTemplates
+ ExtraDisks (with Serial)
+ ExtraVolumes
+ diskSetupScript"] + C["Multi-VM
Network, TPS
+ MultiVMWorkload
(Roles, UserdataForRole)
+ Service + VMCount"] A --> B --> C ``` @@ -534,17 +561,30 @@ Start simple. Add complexity only when the workload needs it. Before submitting a new workload, verify: - [ ] Workload struct embeds `BaseWorkload` -- [ ] Constructor follows the `NewXWorkload(cfg, sshUser, sshPassword, sshKeys)` signature -- [ ] `Name()` returns a lowercase, single-word identifier +- [ ] Constructor follows the `NewXWorkload(cfg, sshUser, sshPassword, sshKeys)` signature (or extended signature for storage / namespace-aware workloads) +- [ ] `Name()` returns a lowercase, hyphen-or-single-word identifier - [ ] `CloudInitUserdata()` calls `w.BuildCloudConfig()` (not `cloudinit.BuildCloudConfig()`) -- [ ] Packages used are available in Fedora's default repos +- [ ] Packages used are available in Fedora's default repos (or pre-installed in the golden image, if applicable) - [ ] Systemd unit has `Restart=always` and `WantedBy=multi-user.target` - [ ] Tests cover: Name, packages, systemd unit content, valid YAML, VMResources, defaults for optional methods -- [ ] Registered in `DefaultRegistry()` with a factory function -- [ ] Added to `AllWorkloadNames` (alphabetical order) +- [ ] Registered in `DefaultRegistry()` (`internal/workloads/registry.go`) with a factory function - [ ] Existing registry/orchestration test counts updated - [ ] `go test ./...` passes - [ ] `go test -race ./...` passes - [ ] `--dry-run` produces a valid VM spec +**If multi-VM:** + +- [ ] Implements `Roles() []string` +- [ ] Implements `UserdataForRole(role, namespace) (string, error)` +- [ ] `VMCount()` returns `Config.VMCount * len(Roles())` +- [ ] `ServiceSpec().Spec.Selector` includes `virtwork/role: ` and `app.kubernetes.io/component: ` +- [ ] Client userdata builds the server DNS as `..svc.cluster.local` + +**If storage-backed:** + +- [ ] `DataVolumeTemplates()` returns templates with stable base names (orchestrator suffixes with VM name) +- [ ] `ExtraDisks()` sets the `Serial` field on each Disk +- [ ] Cloud-init runs `diskSetupScript(serial, mountPoint)` from `runcmd` before any service that uses the mount + See [docs/development.md](../development.md) for the reference version of the "Adding a New Workload" checklist. diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md index ec8d4ba..be00b40 100644 --- a/docs/implementation-plan.md +++ b/docs/implementation-plan.md @@ -1,5 +1,17 @@ # Virtwork Implementation Plan +> **Historical snapshot — original phased build (phases 0–12).** +> +> This document describes the original incremental plan executed to build virtwork through phase 12. It is preserved as a record of the layered design and as context for why packages are organized the way they are. **Subsequent additions are not reflected here.** For the current state, see: +> +> - Chaos workloads (chaos-disk, chaos-network, chaos-process) → [chaos-workloads.md](chaos-workloads.md) +> - TPS workload → [architecture.md](architecture.md) and [guide/02-deploying-workloads.md](guide/02-deploying-workloads.md) +> - Structured logging (`internal/logging`) → [architecture.md](architecture.md) and [development.md](development.md) +> - Golden container disk image → [../build/golden-image/README.md](../build/golden-image/README.md) +> - Audit schema (5 tables) → [audit-schema.md](audit-schema.md) +> - Complete configuration reference → [configuration.md](configuration.md) +> - OpenShift deployment deep-dive → [deployment.md](deployment.md) + This document breaks the [design plan](openshift-virtualization-workload-automation.md) into incremental phases. Each phase produces a testable, committable increment that builds on the previous one. See [architecture.md](architecture.md) for the high-level architecture and diagrams. --- diff --git a/docs/openshift-virtualization-workload-automation.md b/docs/openshift-virtualization-workload-automation.md index 0141049..3e757b3 100644 --- a/docs/openshift-virtualization-workload-automation.md +++ b/docs/openshift-virtualization-workload-automation.md @@ -1,5 +1,15 @@ # Plan: OpenShift Virtualization Workload Automation ("virtwork") +> **Historical snapshot — original design rationale.** +> +> This document captured the motivation and design choices at project inception, when no application code existed yet. It is preserved as context for *why* the layered architecture, the deploy-and-exit model, and the systemd-inside-VM workload lifecycle were chosen. **Implementation has since evolved.** For the current state, see: +> +> - Architecture and design decisions → [architecture.md](architecture.md) +> - Built-in workloads (now nine total) → [README.md](../README.md#workloads), [architecture.md](architecture.md), [chaos-workloads.md](chaos-workloads.md) +> - Developer guide and patterns → [development.md](development.md) +> - Configuration reference → [configuration.md](configuration.md) +> - OpenShift deployment → [deployment.md](deployment.md) + ## Context This codebase (`virtwork`) is a Go CLI tool for creating VMs on an OpenShift cluster with OpenShift Virtualization (CNV) already installed. The goal is to run continuous, varied workloads (CPU, memory, database, network, disk I/O) inside those VMs so that monitoring tools (Prometheus, Grafana, etc.) have realistic metrics to observe. The codebase currently has a `go.mod` and documentation — no application code yet.