From 5f296bc6a0c5e8433e0a4e0c23d9518d5be3be0b Mon Sep 17 00:00:00 2001
From: Melvin Hillsman <mhillsma@redhat.com>
Date: Fri, 22 May 2026 09:52:10 -0500
Subject: [PATCH 01/17] docs(readme): expand workload table to nine entries and
 refresh structure

The workload table only listed five workloads. Expand it to all nine
grouped by purpose (core load generators, multi-VM benchmarks, chaos
engineering) and call out the actual 11-VM default for a no-flag run.

Update the architecture summary and project structure tree to include
internal/logging, build/golden-image, the new docs files, and the
MultiVMWorkload interface.

Add pointers from the configuration, audit, and deployment sections to
their respective deep-dive documents, and add a Makefile shortcut block
since users land on the README and otherwise miss `make ci`/`make verify`.

Signed-off-by: Melvin Hillsman <mhillsma@redhat.com>
---
 README.md | 71 +++++++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 61 insertions(+), 10 deletions(-)

diff --git a/README.md b/README.md
index 8212049..86cc688 100644
--- a/README.md
+++ b/README.md
@@ -100,13 +100,31 @@ virtwork cleanup --run-id <uuid>
 
 ## Workloads
 
+virtwork ships nine built-in workloads, grouped by purpose. With `--vm-count 1` and no `--workloads` filter, a full run creates **11 VMs**: seven single-VM workloads plus two server/client pairs (network and tps).
+
+**Core load generators** — saturate one resource per VM:
+
 | Workload | VMs | Description | Tools |
 |----------|-----|-------------|-------|
-| **cpu** | N (configurable) | Continuous CPU stress | `stress-ng --cpu 0 --cpu-method all` |
-| **memory** | N (configurable) | Memory pressure at 80% | `stress-ng --vm 1 --vm-bytes 80%` |
-| **database** | N (configurable) | PostgreSQL with pgbench loop | `pgbench -c 10 -j 2 -T 300` |
-| **network** | N×2 (server + client pairs) | Bidirectional throughput | `iperf3 --bidir` |
-| **disk** | N (configurable) | Mixed random and sequential I/O | `fio` with multiple profiles |
+| **cpu** | N | Continuous CPU stress | `stress-ng --cpu 0 --cpu-method all` |
+| **memory** | N | Memory pressure at 80% | `stress-ng --vm 1 --vm-bytes 80%` |
+| **disk** | N | Mixed random and sequential I/O on a data disk | `fio` with multiple profiles |
+| **database** | N | PostgreSQL with pgbench loop on a data disk | `pgbench -c 10 -j 2 -T 300` |
+
+**Multi-VM benchmarks** — server/client pairs coordinated via a ClusterIP Service:
+
+| Workload | VMs | Description | Tools |
+|----------|-----|-------------|-------|
+| **network** | N × 2 | Bidirectional throughput | `iperf3 --bidir` on port 5201 |
+| **tps** | N × 2 | TCP request/response + HTTP file transfer | `netperf` (12865/12866) + Python HTTP server (8080) |
+
+**Chaos engineering** — inject failures inside the VM to test resilience. ⚠️ See [docs/chaos-workloads.md](docs/chaos-workloads.md) before deploying:
+
+| Workload | VMs | Description | Tools |
+|----------|-----|-------------|-------|
+| **chaos-disk** | N | Fill the data disk to a target percentage, release, repeat | `fallocate`, `dd` |
+| **chaos-network** | N | Inject latency and packet loss on egress | `tc` + `netem` |
+| **chaos-process** | N | Randomly send signals to non-essential processes | shell + `ps`/`kill` |
 
 All workloads run as systemd services inside the VMs, surviving reboots and auto-restarting on failure.
 
@@ -118,7 +136,7 @@ Deploy VMs with workloads.
 
 ```
 Flags:
-      --workloads strings          Workloads to deploy (default [cpu,database,disk,memory,network])
+      --workloads strings          Workloads to deploy (default: all nine — chaos-disk, chaos-network, chaos-process, cpu, database, disk, memory, network, tps)
       --vm-count int               Number of VMs per workload (default 1)
       --cpu-cores int              CPU cores per VM
       --memory string              Memory per VM (e.g., 2Gi)
@@ -163,6 +181,8 @@ virtwork uses a priority chain for configuration (highest to lowest):
 3. YAML config file (`--config`)
 4. Defaults
 
+The tables below cover the common surface. For a complete reference of every flag, environment variable, YAML key, and per-workload parameter, see [docs/configuration.md](docs/configuration.md).
+
 ### Environment Variables
 
 | Variable | Description |
@@ -205,6 +225,8 @@ The audit database records execution parameters, timestamps, workload details, V
 
 No SSH credentials are stored — only a boolean indicating whether SSH authentication was configured.
 
+For the full schema (five tables with relationships and column descriptions) and common query patterns, see [docs/audit-schema.md](docs/audit-schema.md).
+
 ```bash
 # Disable audit tracking
 virtwork run --no-audit
@@ -246,6 +268,8 @@ When no SSH flags are provided, no user account is configured in the VMs.
 
 virtwork can run as a pod on the cluster using the provided Kustomize manifests in `deploy/`. The deployment manifest uses a semantic version tag (e.g., `v0.0.1`) to pin the image for reproducible deployments. Update the version tag in `deploy/deployment.yaml` when upgrading. For development or testing, you can use `quay.io/opdev/virtwork:latest`.
 
+The section below is a quick reference. For a manifest-by-manifest deep-dive — resource topology, RBAC scope, image pinning policy, sizing, audit-DB persistence — see [docs/deployment.md](docs/deployment.md).
+
 ### Deploy with Kustomize
 
 ```bash
@@ -293,9 +317,11 @@ The codebase follows a strict layered architecture where each layer depends only
 
 ```
 Layer 4 — Orchestration     cmd/virtwork, cleanup, audit
-Layer 3 — Workload Defs     workloads (interface, cpu, memory, database, network, disk, registry)
+Layer 3 — Workload Defs     workloads (Workload + MultiVMWorkload interfaces, registry,
+                            cpu, memory, disk, database, network, tps, chaos-disk,
+                            chaos-network, chaos-process)
 Layer 2 — K8s Abstractions  vm, resources, wait
-Layer 1 — Infrastructure    config, cluster, cloudinit
+Layer 1 — Infrastructure    config, cluster, cloudinit, logging
 Layer 0 — Definitions       constants
 ```
 
@@ -313,15 +339,18 @@ virtwork/
 │   ├── config/                    # Viper-based config priority chain
 │   ├── cluster/                   # controller-runtime client init
 │   ├── cloudinit/                 # Cloud-config YAML builder
+│   ├── logging/                   # Structured logger (log/slog wrapper)
 │   ├── vm/                        # VM spec construction + CRUD + retry
 │   ├── resources/                 # Namespace + Service + Secret helpers
 │   ├── wait/                      # VMI readiness polling
 │   ├── cleanup/                   # Label-based teardown (VMs, Services, Secrets)
 │   ├── audit/                     # SQLite audit tracking (Auditor interface, schema, records)
-│   ├── workloads/                 # Workload interface + 5 implementations + registry
+│   ├── workloads/                 # Workload + MultiVMWorkload interfaces, 9 implementations, registry
 │   └── testutil/                  # Shared test helpers for integration + E2E
 ├── tests/
 │   └── e2e/                       # E2E acceptance tests (//go:build e2e)
+├── build/
+│   └── golden-image/              # Optional Fedora container disk with pre-installed tools
 ├── deploy/                        # Kustomize manifests for OpenShift deployment
 │   ├── kustomization.yaml
 │   ├── namespace.yaml
@@ -332,11 +361,22 @@ virtwork/
 │   ├── pvc.yaml
 │   └── deployment.yaml
 ├── Dockerfile                     # Multi-stage build (Debian builder + UBI9 runtime)
+├── Dockerfile.ci                  # CI variant of the runtime image
 ├── entrypoint.sh                  # Container entrypoint (auto-run or sleep)
+├── Makefile                       # CI targets (test, vet, lint, build, ci, verify)
 ├── docs/
+│   ├── README.md                  # Documentation index
 │   ├── architecture.md            # Layered architecture and diagrams
 │   ├── development.md             # Developer guide
-│   └── implementation-plan.md     # Phased build plan
+│   ├── configuration.md           # Complete config reference (flags, env, YAML)
+│   ├── deployment.md              # OpenShift deployment deep-dive
+│   ├── audit-schema.md            # SQLite audit schema reference
+│   ├── chaos-workloads.md         # Chaos engineering workload guide
+│   ├── virtwork-vs-kube-burner.md # Positioning vs kube-burner
+│   ├── guide/                     # Hands-on guides (overview, deploying, adding workloads)
+│   ├── implementation-plan.md     # Historical: original phased build plan
+│   └── openshift-virtualization-workload-automation.md  # Historical: original design rationale
+├── OWNERS
 ├── go.mod
 └── go.sum
 ```
@@ -371,6 +411,17 @@ go test -tags "integration e2e" ./...
 go build -o virtwork ./cmd/virtwork
 ```
 
+### Makefile shortcuts
+
+```bash
+make help          # list all targets
+make test          # unit tests with race detector and coverage
+make ci            # vet + test + build (no cluster required)
+make verify        # fmt + vet + lint + test (full pre-commit)
+make build         # build binary to bin/virtwork
+make container-build  # build the OCI image locally
+```
+
 See [docs/development.md](docs/development.md) for the full developer guide, including instructions for adding new workloads.
 
 ## License

From 0291d85572b37366332018b781d5fbd224d922ef Mon Sep 17 00:00:00 2001
From: Melvin Hillsman <mhillsma@redhat.com>
Date: Fri, 22 May 2026 09:52:56 -0500
Subject: [PATCH 02/17] docs(arch): redraw layered diagram with logging and all
 nine workloads

The mermaid layered architecture diagram only showed five workloads and
omitted the internal/logging package added when structured logging
replaced fmt.Fprintf calls. Redraw it to include:

- All nine workloads grouped beneath the registry (cpu, memory, disk,
  database in core; network and tps as multi-VM via the new
  MultiVMWorkload interface; chaos-disk, chaos-network, chaos-process)
- The MultiVMWorkload interface as its own node, since network and tps
  implement Roles() and UserdataForRole() on top of Workload
- internal/logging in Layer 1, with edges from cmd/virtwork and from
  internal/wait (the two current consumers)

Signed-off-by: Melvin Hillsman <mhillsma@redhat.com>
---
 docs/architecture.md | 26 ++++++++++++++++++++++----
 1 file changed, 22 insertions(+), 4 deletions(-)

diff --git a/docs/architecture.md b/docs/architecture.md
index 3099637..d5251af 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -23,11 +23,16 @@ graph TD
     subgraph "Layer 3 — Workload Definitions"
         REGISTRY["internal/workloads/registry.go\nregistry + lookup"]
         IFACE["internal/workloads/workload.go\nWorkload interface"]
+        MULTI["internal/workloads/workload.go\nMultiVMWorkload interface\n(Roles, UserdataForRole)"]
         CPU["internal/workloads/cpu.go\nstress-ng CPU"]
         MEM["internal/workloads/memory.go\nstress-ng VM memory"]
+        DISK["internal/workloads/disk.go\nfio profiles"]
         DB["internal/workloads/database.go\nPostgreSQL + pgbench"]
         NET["internal/workloads/network.go\niperf3 server/client"]
-        DISK["internal/workloads/disk.go\nfio profiles"]
+        TPS["internal/workloads/tps.go\nnetperf TCP_RR + HTTP"]
+        CDISK["internal/workloads/chaos_disk.go\nfill/release loop"]
+        CNET["internal/workloads/chaos-network.go\ntc/netem latency + loss"]
+        CPROC["internal/workloads/chaos_process.go\nrandom signal killer"]
     end
 
     subgraph "Layer 2 — K8s Abstractions"
@@ -40,6 +45,7 @@ graph TD
         CLUSTER["internal/cluster/cluster.go\ncontroller-runtime client init"]
         CONFIG["internal/config/config.go\nViper config"]
         CLOUDINIT["internal/cloudinit/cloudinit.go\ncloud-config YAML builder"]
+        LOGGING["internal/logging/logging.go\nstructured slog logger"]
     end
 
     subgraph "Layer 0 — Definitions"
@@ -54,6 +60,7 @@ graph TD
     CMD --> WAIT
     CMD --> CLEANUP
     CMD --> AUDIT
+    CMD --> LOGGING
 
     AUDIT --> CONFIG
     AUDIT --> CONST
@@ -64,15 +71,25 @@ graph TD
 
     REGISTRY --> CPU
     REGISTRY --> MEM
+    REGISTRY --> DISK
     REGISTRY --> DB
     REGISTRY --> NET
-    REGISTRY --> DISK
+    REGISTRY --> TPS
+    REGISTRY --> CDISK
+    REGISTRY --> CNET
+    REGISTRY --> CPROC
 
     CPU --> IFACE
     MEM --> IFACE
-    DB --> IFACE
-    NET --> IFACE
     DISK --> IFACE
+    DB --> IFACE
+    CDISK --> IFACE
+    CNET --> IFACE
+    CPROC --> IFACE
+
+    NET --> MULTI
+    TPS --> MULTI
+    MULTI --> IFACE
 
     VM --> CLUSTER
     VM --> CONST
@@ -83,6 +100,7 @@ graph TD
 
     WAIT --> CLUSTER
     WAIT --> CONST
+    WAIT --> LOGGING
 
     CLUSTER --> CONST
     CONFIG --> CONST

From 130a00227ec3447363f47e0afcd3624f679429e7 Mon Sep 17 00:00:00 2001
From: Melvin Hillsman <mhillsma@redhat.com>
Date: Fri, 22 May 2026 09:53:41 -0500
Subject: [PATCH 03/17] docs(arch): expand workload class diagram with
 MultiVMWorkload, tps, chaos

The class diagram covered only the original five workloads and did not
show MultiVMWorkload at all, even though two workloads (network, tps)
implement it. Expand the diagram to include:

- MultiVMWorkload interface (Roles, UserdataForRole) sitting above
  Workload, with NetworkWorkload and TPSWorkload realizing it
- TPSWorkload, ChaosDiskWorkload, ChaosNetworkWorkload, ChaosProcessWorkload
  with their concrete Name(), userdata characteristics, and any extra
  storage / service surface
- Notes on virtio Serial fields for disk-backed workloads, since the
  in-VM disk discovery now goes through /dev/disk/by-id/virtio-<serial>

NetworkWorkload was previously shown with invented ServerUserdata()/
ClientUserdata() methods; replace with the actual UserdataForRole entry
point and accurate service port (5201). TPS service shows the three
ports actually exposed: netperf-ctrl 12865, netperf-data 12866, http-data 8080.

Signed-off-by: Melvin Hillsman <mhillsma@redhat.com>
---
 docs/architecture.md | 63 +++++++++++++++++++++++++++++++++++++-------
 1 file changed, 53 insertions(+), 10 deletions(-)

diff --git a/docs/architecture.md b/docs/architecture.md
index d5251af..89f46de 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -249,6 +249,12 @@ classDiagram
         +VMCount() int
     }
 
+    class MultiVMWorkload {
+        <<interface>>
+        +Roles() []string
+        +UserdataForRole(role, namespace) (string, error)
+    }
+
     class BaseWorkload {
         +Config WorkloadConfig
         +SSHUser string
@@ -267,43 +273,80 @@ classDiagram
     class CPUWorkload {
         +Name() "cpu"
         +CloudInitUserdata() stress-ng --cpu config
-        +VMResources() cpu/memory from config
     }
 
     class MemoryWorkload {
         +Name() "memory"
         +CloudInitUserdata() stress-ng --vm config
-        +VMResources() cpu/memory from config
+    }
+
+    class DiskWorkload {
+        +Name() "disk"
+        +CloudInitUserdata() fio profiles
+        +DataVolumeTemplates() blank DV for /mnt/data
+        +ExtraDisks() data disk with virtio Serial
     }
 
     class DatabaseWorkload {
         +Name() "database"
         +CloudInitUserdata() postgresql + pgbench
         +DataVolumeTemplates() blank DV for /var/lib/pgsql/data
+        +ExtraDisks() data disk with virtio Serial
     }
 
     class NetworkWorkload {
         +Namespace string
         +Name() "network"
-        +VMCount() count * 2 (server+client pairs)
+        +VMCount() count * 2
+        +Roles() ["server", "client"]
+        +UserdataForRole(role, ns) iperf3 server or client
         +RequiresService() true
-        +ServerUserdata() iperf3 -s
-        +ClientUserdata() iperf3 -c
-        +ServiceSpec() ClusterIP for server
+        +ServiceSpec() ClusterIP virtwork-iperf3-server :5201
     }
 
-    class DiskWorkload {
-        +Name() "disk"
-        +CloudInitUserdata() fio profiles
+    class TPSWorkload {
+        +Namespace string
+        +Name() "tps"
+        +VMCount() count * 2
+        +Roles() ["server", "client"]
+        +UserdataForRole(role, ns) netperf + HTTP server / client loop
+        +RequiresService() true
+        +ServiceSpec() ClusterIP virtwork-tps-server :12865/:12866/:8080
+        +Params file-size, iterations, duration
+    }
+
+    class ChaosDiskWorkload {
+        +Name() "chaos-disk"
+        +CloudInitUserdata() fallocate fill / rm release loop
         +DataVolumeTemplates() blank DV for /mnt/data
+        +ExtraDisks() data disk with virtio Serial
     }
 
+    class ChaosNetworkWorkload {
+        +Latency int (ms)
+        +PacketLoss float64 (%)
+        +Name() "chaos-network"
+        +CloudInitUserdata() tc qdisc netem delay + loss
+    }
+
+    class ChaosProcessWorkload {
+        +Name() "chaos-process"
+        +CloudInitUserdata() random kill loop with excluded patterns
+    }
+
+    Workload <|-- MultiVMWorkload
     Workload <|.. BaseWorkload
     BaseWorkload <|-- CPUWorkload
     BaseWorkload <|-- MemoryWorkload
+    BaseWorkload <|-- DiskWorkload
     BaseWorkload <|-- DatabaseWorkload
     BaseWorkload <|-- NetworkWorkload
-    BaseWorkload <|-- DiskWorkload
+    BaseWorkload <|-- TPSWorkload
+    BaseWorkload <|-- ChaosDiskWorkload
+    BaseWorkload <|-- ChaosNetworkWorkload
+    BaseWorkload <|-- ChaosProcessWorkload
+    MultiVMWorkload <|.. NetworkWorkload
+    MultiVMWorkload <|.. TPSWorkload
 ```
 
 `BaseWorkload` is an embedded struct that provides default implementations for optional interface methods. Concrete workloads embed `BaseWorkload` and override only the methods they need — idiomatic Go composition over inheritance.

From dd5c16ed08c6df918c377ec2d47f5898c9f98e3c Mon Sep 17 00:00:00 2001
From: Melvin Hillsman <mhillsma@redhat.com>
Date: Fri, 22 May 2026 09:56:11 -0500
Subject: [PATCH 04/17] docs(arch): refresh comparison, concurrency, and design
 decision tables
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Update the Workload Comparison table to nine rows including tps and the
three chaos workloads. Show actual service names and ports (network ->
virtwork-iperf3-server :5201; tps -> virtwork-tps-server :12865/12866/8080),
correct package lists per workload, and the tools each one runs.

Add internal/logging to the Concurrency Model table — it is stateless and
shared across goroutines that call it.

Add four rows to Key Design Decisions:

- In-VM disk discovery via virtio Serial. /dev/vdX ordering is not stable
  across reboots; the diskSetupScript helper handles wait/format/mount/fstab.
- DataVolume per-VM name suffixing via namespaceDataVolumes, since
  DataVolume names are namespace-scoped and would collide across VMs.
- Structured logging via internal/logging (slog), set by --verbose.
- Chaos workload safety — opt-in by name with namespace isolation as the
  boundary, no kill switch. Cross-link chaos-workloads.md.

Also cross-link audit-schema.md from the audit storage row.

Signed-off-by: Melvin Hillsman <mhillsma@redhat.com>
---
 docs/architecture.md | 15 ++++++++++++---
 1 file changed, 12 insertions(+), 3 deletions(-)

diff --git a/docs/architecture.md b/docs/architecture.md
index 89f46de..7f87c41 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -168,6 +168,7 @@ graph LR
 | `internal/config` | No | One-time Viper load at startup |
 | `internal/cloudinit` | No | Pure string/YAML generation |
 | `internal/cluster` | No | One-time client init at startup |
+| `internal/logging` | No | Stateless `slog` wrapper; safe to share across goroutines that call it |
 | `internal/vm` | Yes | CRUD operations run in errgroup goroutines; retry loops use `time.Sleep` |
 | `internal/resources` | Yes | Namespace/Service/Secret creation can run concurrently |
 | `internal/wait` | Yes | Concurrent VMI polling via errgroup; uses `time.Sleep` between polls |
@@ -359,9 +360,13 @@ classDiagram
 |----------|----------|-------------|-------------|----------|---------------|
 | CPU | N (configurable) | No | No | stress-ng | `stress-ng --cpu 0 --cpu-method all` |
 | Memory | N (configurable) | No | No | stress-ng | `stress-ng --vm 1 --vm-bytes 80% --vm-method all` |
-| Database | N (configurable) | Yes (`/var/lib/pgsql/data`) | No | postgresql-server | `pgbench -c 10 -j 2 -T 300` loop |
-| Network | N×2 (server + client pairs) | No | Yes (ClusterIP) | iperf3 | `iperf3 -s` / `iperf3 -c ... --bidir` |
 | Disk | N (configurable) | Yes (`/mnt/data`) | No | fio | Mixed R/W + sequential write profiles |
+| Database | N (configurable) | Yes (`/var/lib/pgsql/data`) | No | postgresql-server | `pgbench -c 10 -j 2 -T 300` loop |
+| Network | N × 2 (server + client) | No | Yes — `virtwork-iperf3-server` :5201 | iperf3 | `iperf3 -s` / `iperf3 -c ... --bidir` |
+| TPS | N × 2 (server + client) | No | Yes — `virtwork-tps-server` :12865 / :12866 / :8080 | netperf, python3 | `netperf -t TCP_RR` + curl HTTP file fetch loop |
+| Chaos-disk | N (configurable) | Yes (`/mnt/data`) | No | (golden image: `fallocate`, `dd`) | Fill to target percent, sleep, release, repeat |
+| Chaos-network | N (configurable) | No | No | iproute-tc (+ `sch_netem` kernel module) | `tc qdisc add ... netem delay 100ms loss 5%` |
+| Chaos-process | N (configurable) | No | No | procps-ng | Random `kill -SIGTERM <pid>` of non-essential processes every 30s |
 
 ---
 
@@ -468,9 +473,13 @@ Viper's built-in priority chain handles this natively when bound to Cobra flags:
 | Network VM scaling | `VMCount() = count * 2` | Honors `--vm-count` to create N server/client pairs instead of a single hardcoded pair. |
 | Cloud-init Secrets | `CloudInitSecretName` → `UserDataSecretRef` | For large userdata, stores cloud-init in a K8s Secret instead of inline in the VM spec. |
 | Cleanup error semantics | Sequential per-resource deletion with error accumulation | Different from create-time error handling (which is fail-fast). Cleanup continues on individual failures. |
-| Audit storage | SQLite (`virtwork.db`) with `Auditor` interface | Local file, zero infrastructure. `NoOpAuditor` when disabled. WAL mode for concurrent safety. |
+| Audit storage | SQLite (`virtwork.db`) with `Auditor` interface | Local file, zero infrastructure. `NoOpAuditor` when disabled. WAL mode for concurrent safety. See [audit-schema.md](audit-schema.md) for the full schema. |
 | Run-to-cleanup linking | `virtwork/run-id` K8s label + `linked_run_ids` JSON array | Labels survive across CLI invocations. JSON array is PostgreSQL JSONB compatible. |
 | Audit credential policy | No SSH credentials stored | Only `ssh_auth_configured` boolean tracked. Security by design. |
+| In-VM disk discovery | `/dev/disk/by-id/virtio-<serial>` via the `Serial` field on KubeVirt `Disk` | `/dev/vdX` device ordering is not stable across VM reboots or migrations; the virtio serial provides a deterministic symlink. The shared `diskSetupScript` helper (`internal/workloads/workload.go`) waits for the symlink, formats if empty, mounts, and writes `/etc/fstab`. |
+| DataVolume names per VM | DV template names are suffixed with the VM name via `namespaceDataVolumes` (`cmd/virtwork/main.go`) | DataVolume names are namespace-scoped; deploying multiple VMs of the same workload would otherwise collide on the template name. |
+| Structured logging | `log/slog` JSON via `internal/logging.NewLogger(out, verbose)` | Machine-parseable logs for pipeline consumption; `--verbose` flips the level from `INFO` to `DEBUG`. New code uses the logger; `fmt.Fprintf` calls were removed from `cmd/virtwork/main.go`. |
+| Chaos workload safety | Opt-in by name, namespace-scoped destructive behavior, no platform-level kill switch | Chaos workloads can fill a data PVC, shape egress traffic, or kill processes — all confined to the VM they run in. Namespace isolation is the safety boundary. See [chaos-workloads.md](chaos-workloads.md) for risk and operational guidance. |
 
 ---
 

From e1a26f4cd8d58933056bebc79c6e2b30fb26bbe5 Mon Sep 17 00:00:00 2001
From: Melvin Hillsman <mhillsma@redhat.com>
Date: Fri, 22 May 2026 09:57:07 -0500
Subject: [PATCH 05/17] docs(arch): remove engineering-journals reference and
 refresh project tree

The architecture.md project structure listed docs/engineering-journals/
but the directory does not exist, and the CLAUDE.md guideline explicitly
forbids committing engineering journals. Remove the entry rather than
recreate a confused state.

Refresh the project tree to match reality:

- Add internal/logging and the four newly-existing workload files
  (tps.go, chaos_disk.go, chaos-network.go, chaos_process.go)
- Update internal/workloads/workload.go description to mention
  MultiVMWorkload and the diskSetupScript helper
- Update registry.go description to "nine entries"
- Add build/golden-image/, the new reference docs (configuration.md,
  deployment.md, audit-schema.md, chaos-workloads.md), docs/README.md,
  and docs/guide/
- Add Dockerfile.ci, Makefile, OWNERS for completeness

Signed-off-by: Melvin Hillsman <mhillsma@redhat.com>
---
 docs/architecture.md | 34 +++++++++++++++++++++++++++-------
 1 file changed, 27 insertions(+), 7 deletions(-)

diff --git a/docs/architecture.md b/docs/architecture.md
index 7f87c41..a586519 100644
--- a/docs/architecture.md
+++ b/docs/architecture.md
@@ -499,6 +499,8 @@ virtwork/
 │   │   └── cluster.go             # controller-runtime client init + scheme registration
 │   ├── cloudinit/
 │   │   └── cloudinit.go           # Cloud-config YAML builder
+│   ├── logging/
+│   │   └── logging.go             # Structured slog logger (verbose -> DEBUG)
 │   ├── vm/
 │   │   └── vm.go                  # VM spec construction + typed CRUD + retry
 │   ├── resources/
@@ -512,25 +514,43 @@ virtwork/
 │   │   ├── schema.go              # DDL for 5 audit tables + indexes
 │   │   └── records.go             # WorkloadRecord, VMRecord, ResourceRecord, EventRecord
 │   ├── workloads/
-│   │   ├── workload.go            # Workload interface + BaseWorkload
-│   │   ├── registry.go            # Registry map + lookup
+│   │   ├── workload.go            # Workload + MultiVMWorkload interfaces, BaseWorkload, diskSetupScript
+│   │   ├── registry.go            # Registry map + DefaultRegistry (nine entries)
 │   │   ├── cpu.go                 # stress-ng CPU continuous workload
 │   │   ├── memory.go              # stress-ng VM memory pressure workload
+│   │   ├── disk.go                # fio mixed I/O profiles
 │   │   ├── database.go            # PostgreSQL + pgbench loop
-│   │   ├── network.go             # iperf3 server/client pair
-│   │   └── disk.go                # fio mixed I/O profiles
+│   │   ├── network.go             # iperf3 server/client pair (MultiVMWorkload)
+│   │   ├── tps.go                 # netperf TCP_RR + HTTP file transfer (MultiVMWorkload)
+│   │   ├── chaos_disk.go          # Fill-and-release disk pressure loop
+│   │   ├── chaos-network.go       # tc/netem latency + loss injection
+│   │   └── chaos_process.go       # Random kill loop with excluded process patterns
 │   └── testutil/
 │       ├── testutil.go            # Shared test helpers (namespace, connect, cleanup)
 │       └── binary.go              # Binary build/run helpers for E2E
 ├── tests/
 │   └── e2e/                       # E2E acceptance tests (//go:build e2e)
+├── build/
+│   └── golden-image/              # Optional Fedora container disk with pre-installed tools
+├── deploy/                        # Kustomize manifests for OpenShift deployment
 ├── docs/
+│   ├── README.md                  # Documentation index
 │   ├── architecture.md            # This file
 │   ├── development.md             # Developer guide
-│   ├── implementation-plan.md     # Phased build plan
-│   ├── openshift-virtualization-workload-automation.md  # Design plan
-│   └── engineering-journals/      # Per-phase development journals
+│   ├── configuration.md           # Complete config reference
+│   ├── deployment.md              # OpenShift deployment deep-dive
+│   ├── audit-schema.md            # SQLite audit schema reference
+│   ├── chaos-workloads.md         # Chaos engineering workload guide
+│   ├── virtwork-vs-kube-burner.md # Positioning vs kube-burner
+│   ├── guide/                     # Hands-on guides (overview, deploying, adding workloads)
+│   ├── implementation-plan.md     # Historical: original phased build plan
+│   └── openshift-virtualization-workload-automation.md  # Historical: original design rationale
+├── Dockerfile                     # Multi-stage build (Debian builder + UBI9 runtime)
+├── Dockerfile.ci                  # CI variant of the runtime image
+├── entrypoint.sh
+├── Makefile
 ├── go.mod
 ├── go.sum
+├── OWNERS
 └── CLAUDE.md
 ```

From 640dc504690438185b88777890edebc3ed8cfe69 Mon Sep 17 00:00:00 2001
From: Melvin Hillsman <mhillsma@redhat.com>
Date: Fri, 22 May 2026 10:25:57 -0500
Subject: [PATCH 06/17] docs(dev): document logging, multi-VM, storage-backed
 patterns; refresh layout
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The development guide referenced engineering-journals/ (now removed),
described five workloads instead of nine, and missed three patterns that
new contributors need:

- Multi-VM workloads via the MultiVMWorkload interface (Roles,
  UserdataForRole, in-cluster DNS for client-server coordination). The
  canonical references are network.go (simple) and tps.go (configurable
  Params).
- Storage-backed workloads via the virtio Serial field on Disk plus the
  shared diskSetupScript helper that waits for /dev/disk/by-id/virtio-*,
  formats, mounts, and writes /etc/fstab. References: disk.go, database.go,
  chaos_disk.go.
- Structured logging via internal/logging.NewLogger — JSON output, --verbose
  flips INFO to DEBUG. Workload pure-data layer does not log.

Update the registry example to all nine current entries, refresh the
project layout tree (add internal/logging, build/golden-image/, deploy/,
new docs), and add internal/logging to the architecture layers table.

Add a Related Documentation section linking the new reference docs, and
note the DCO Signed-off-by hook in the commit conventions section.

Signed-off-by: Melvin Hillsman <mhillsma@redhat.com>
---
 docs/development.md | 96 +++++++++++++++++++++++++++++++++++++++------
 1 file changed, 85 insertions(+), 11 deletions(-)

diff --git a/docs/development.md b/docs/development.md
index 6b5a13d..4b74a98 100644
--- a/docs/development.md
+++ b/docs/development.md
@@ -421,20 +421,28 @@ internal/           # Application packages (not importable externally)
   config/           # Config struct, Viper priority chain
   cluster/          # controller-runtime client init + scheme
   cloudinit/        # Cloud-config YAML builder
+  logging/          # Structured slog logger (verbose -> DEBUG)
   vm/               # VM spec construction + typed CRUD + retry
   resources/        # Namespace + Service + Secret helpers
   wait/             # VMI readiness polling (errgroup)
   cleanup/          # Label-based teardown (VMs, Services, Secrets)
   audit/            # SQLite audit tracking (Auditor interface, schema, records)
-  workloads/        # Workload interface, 5 implementations, registry
+  workloads/        # Workload + MultiVMWorkload interfaces, nine implementations, registry
   testutil/         # Shared test helpers for integration and E2E tests
 tests/              # Tests requiring external infrastructure
   e2e/              # E2E acceptance tests (//go:build e2e)
+build/
+  golden-image/     # Optional Fedora container disk with pre-installed tools
+deploy/             # Kustomize manifests for OpenShift deployment
 docs/               # Documentation
+  README.md         # Documentation index
   architecture.md   # Layered architecture and mermaid diagrams
-  implementation-plan.md  # Phased build plan
   development.md    # This file
-  engineering-journals/   # Per-phase development journals
+  configuration.md  # Complete configuration reference
+  deployment.md     # OpenShift deployment deep-dive
+  audit-schema.md   # SQLite audit schema reference
+  chaos-workloads.md  # Chaos engineering workload guide
+  guide/            # Hands-on guides (overview, deploying, adding workloads)
 ```
 
 ## Architecture Layers
@@ -444,7 +452,7 @@ The codebase follows a strict layered architecture where each layer depends only
 | Layer | Packages | Goroutines | Purpose |
 |-------|----------|------------|---------|
 | 0 | `constants` | No | Pure values — API coordinates, labels, defaults |
-| 1 | `config`, `cloudinit`, `cluster` | No | Configuration, cloud-init YAML, K8s client init |
+| 1 | `config`, `cloudinit`, `cluster`, `logging` | No | Configuration, cloud-init YAML, K8s client init, structured logging |
 | 2 | `vm`, `resources`, `wait` | Yes | K8s CRUD operations with retry, readiness polling |
 | 3 | `workloads` | No | Pure data producers — cloud-init specs, resource structs |
 | 4 | `cmd/virtwork`, `cleanup`, `audit` | Yes | Orchestration, teardown, and audit tracking |
@@ -531,17 +539,21 @@ func (w *MyWorkload) UserdataForRole(role string, namespace string) (string, err
 
 ### 3. Register the Workload
 
-Add the constructor to `internal/workloads/registry.go`:
+Add the constructor to `internal/workloads/registry.go`. `DefaultRegistry()` currently has nine entries; add yours alongside:
 
 ```go
 func DefaultRegistry() Registry {
     return Registry{
-        "cpu":         func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewCPUWorkload(cfg, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) },
-        "memory":      func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewMemoryWorkload(cfg, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) },
-        "database":    func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewDatabaseWorkload(cfg, opts.DataDiskSize, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) },
-        "network":     func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewNetworkWorkload(cfg, opts.Namespace, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) },
-        "disk":        func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewDiskWorkload(cfg, opts.DataDiskSize, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) },
-        "my-workload": func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewMyWorkload(cfg, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) },
+        "cpu":           func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewCPUWorkload(cfg, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) },
+        "memory":        func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewMemoryWorkload(cfg, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) },
+        "disk":          func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewDiskWorkload(cfg, opts.DataDiskSize, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) },
+        "database":      func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewDatabaseWorkload(cfg, opts.DataDiskSize, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) },
+        "network":       func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewNetworkWorkload(cfg, opts.Namespace, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) },
+        "tps":           func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewTPSWorkload(cfg, opts.Namespace, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) },
+        "chaos-disk":    func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewChaosDiskWorkload(cfg, opts.DataDiskSize, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) },
+        "chaos-network": func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewChaosNetworkWorkload(cfg, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) },
+        "chaos-process": func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewChaosProcessWorkload(cfg, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) },
+        "my-workload":   func(cfg config.WorkloadConfig, opts *RegistryOpts) Workload { return NewMyWorkload(cfg, opts.SSHUser, opts.SSHPassword, opts.SSHAuthorizedKeys) },
     }
 }
 ```
@@ -551,6 +563,55 @@ func DefaultRegistry() Registry {
 - Orchestration BDD tests will fail (total VM count assertions)
 - Update both before considering the feature complete
 
+### Going Further: Multi-VM Workloads
+
+If your workload needs more than one role of VM (e.g., a server and a client), implement the `MultiVMWorkload` interface in addition to `Workload`:
+
+```go
+// In internal/workloads/workload.go
+type MultiVMWorkload interface {
+    Workload
+    Roles() []string
+    UserdataForRole(role string, namespace string) (string, error)
+}
+```
+
+Pattern:
+
+1. Embed `BaseWorkload` and store any per-workload state (e.g., a `Namespace` field for in-cluster DNS).
+2. Override `VMCount()` to return `count * len(Roles())` so the orchestrator creates the right number of VMs per role.
+3. Override `Roles()` to return the role names (e.g., `[]string{"server", "client"}`).
+4. Override `UserdataForRole(role, namespace)` to return role-specific cloud-init. The default `CloudInitUserdata()` typically delegates to `UserdataForRole("server", namespace)`.
+5. Set `RequiresService()` to `true` and provide a `ServiceSpec()` selecting server VMs by the `virtwork/role: server` label that the orchestrator applies automatically.
+6. Clients reach servers via the in-cluster DNS name `<service-name>.<namespace>.svc.cluster.local` — never poll for pod IPs.
+
+The canonical references are `internal/workloads/network.go` (simplest — one port, iperf3) and `internal/workloads/tps.go` (multi-port Service with configurable `Params` for `file-size`, `iterations`, `duration`).
+
+### Going Further: Storage-Backed Workloads
+
+If your workload needs persistent storage inside the VM:
+
+1. Override `DataVolumeTemplates()` to return a CDI `DataVolume` for each volume needed. Use `vm.BuildDataVolumeTemplate(name, size)` from `internal/vm`.
+2. Override `ExtraVolumes()` and `ExtraDisks()` to wire the DataVolume into the VM. **Always set the `Serial` field on the `Disk`** — the in-VM script discovers the device through `/dev/disk/by-id/virtio-<serial>`, which is deterministic across reboots (unlike `/dev/vdX`, which is not).
+3. In your cloud-init userdata, write the shared `diskSetupScript(serial, mountPoint)` helper (from `internal/workloads/workload.go`) as the first script. It waits for the symlink, formats with XFS if empty, mounts, and writes `/etc/fstab` for persistence across reboots.
+4. The orchestrator's `namespaceDataVolumes` helper automatically suffixes DV template names with the VM name to avoid collisions across multiple VMs of the same workload. Your template name should be the un-suffixed base (e.g., `virtwork-chaos-disk-data`).
+
+Reference workloads: `disk.go` (single fio mount), `database.go` (PostgreSQL data dir), `chaos_disk.go` (fill/release loop). All three follow the same pattern.
+
+### Going Further: Structured Logging
+
+The `internal/logging` package provides a shared `*slog.Logger` returned by `NewLogger(w io.Writer, verbose bool)`. Use it instead of `fmt.Fprintf` or `log.Printf` in any code path under `cmd/` or in packages that perform I/O (`internal/wait` is the current example):
+
+```go
+logger := logging.NewLogger(cmd.OutOrStdout(), verbose)
+logger.Info("vm created",
+    slog.String("vm_name", name),
+    slog.String("namespace", ns),
+    slog.String("workload", component))
+```
+
+The output is JSON. `--verbose` flips the level from `INFO` to `DEBUG`. Workload constructors and the pure data layer (`internal/workloads`, `internal/cloudinit`) should not log — they remain pure data producers.
+
 ### 4. Write Tests
 
 Create `internal/workloads/my_workload_test.go` using Ginkgo:
@@ -734,6 +795,19 @@ This project uses [Conventional Commits](https://www.conventionalcommits.org/):
 | `refactor:` | Code restructuring without behavior change |
 | `chore:` | Build, tooling, or maintenance |
 
+Every commit must include a DCO `Signed-off-by` trailer matching the author's identity; a `commit-msg` hook enforces this.
+
+---
+
+## Related Documentation
+
+- [architecture.md](architecture.md) — Layered architecture, mermaid diagrams, key design decisions
+- [configuration.md](configuration.md) — Complete config reference (flags, env vars, YAML keys, ConfigMap)
+- [audit-schema.md](audit-schema.md) — SQLite schema for the audit database
+- [chaos-workloads.md](chaos-workloads.md) — Operator guide for chaos-disk, chaos-network, chaos-process
+- [deployment.md](deployment.md) — OpenShift deployment via Kustomize
+- [guide/03-adding-a-workload.md](guide/03-adding-a-workload.md) — TDD walkthrough that builds a new workload from scratch
+
 ## Idempotency and Safety
 
 - `apierrors.IsAlreadyExists()` responses are treated as success (resource already exists)

From a4460cb95cbbd6b484e8c870c949c9b3d5fd321a Mon Sep 17 00:00:00 2001
From: Melvin Hillsman <mhillsma@redhat.com>
Date: Fri, 22 May 2026 10:26:49 -0500
Subject: [PATCH 07/17] docs(guide): expand 01-overview workload table and code
 tour to nine workloads
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The complexity table listed only five workloads. Expand to all nine,
grouping by overrides-beyond-Name+CloudInit:

- Simplest: cpu, memory, chaos-process, chaos-network
- Medium (storage-backed): disk, database, chaos-disk — all using the
  virtio Serial + diskSetupScript pattern
- Most complex (multi-VM): network and tps — both implementing
  MultiVMWorkload with Roles and UserdataForRole

Update the "Finding Your Way Around the Code" table to include the new
workload files, the diskSetupScript helper, the namespaceDataVolumes
orchestration helper, the internal/logging package, the build/golden-image
directory, and a pointer to audit-schema.md.

Signed-off-by: Melvin Hillsman <mhillsma@redhat.com>
---
 docs/guide/01-overview.md | 25 +++++++++++++++++--------
 1 file changed, 17 insertions(+), 8 deletions(-)

diff --git a/docs/guide/01-overview.md b/docs/guide/01-overview.md
index fcdfdec..e433156 100644
--- a/docs/guide/01-overview.md
+++ b/docs/guide/01-overview.md
@@ -208,17 +208,23 @@ func (b *BaseWorkload) RequiresService() bool              { return false }
 // ... etc
 ```
 
-Concrete workloads embed `BaseWorkload` and only override what they need. This creates a natural complexity spectrum:
+Concrete workloads embed `BaseWorkload` and only override what they need. This creates a natural complexity spectrum across all nine built-in workloads:
 
 | Workload | Overrides Beyond Name + CloudInit | Complexity |
 |----------|-----------------------------------|------------|
 | **CPU** | Nothing | Simplest |
 | **Memory** | Nothing | Simplest |
-| **Database** | `DataVolumeTemplates`, `ExtraDisks`, `ExtraVolumes` | Medium |
-| **Disk** | `DataVolumeTemplates`, `ExtraDisks`, `ExtraVolumes` | Medium |
-| **Network** | `VMCount`, `RequiresService`, `ServiceSpec`, plus `MultiVMWorkload` interface | Most complex |
+| **Chaos-process** | Nothing (systemd unit defaults baked in) | Simplest |
+| **Chaos-network** | Nothing in Go (`Latency`, `PacketLoss` baked into struct) | Simple |
+| **Disk** | `DataVolumeTemplates`, `ExtraDisks`, `ExtraVolumes` (virtio Serial + `diskSetupScript`) | Medium |
+| **Database** | `DataVolumeTemplates`, `ExtraDisks`, `ExtraVolumes` (virtio Serial + `diskSetupScript`) | Medium |
+| **Chaos-disk** | `DataVolumeTemplates`, `ExtraDisks`, `ExtraVolumes` (virtio Serial + `diskSetupScript`) | Medium |
+| **Network** | `VMCount`, `Roles`, `UserdataForRole`, `RequiresService`, `ServiceSpec` (implements `MultiVMWorkload`) | Most complex |
+| **TPS** | `VMCount`, `Roles`, `UserdataForRole`, `RequiresService`, `ServiceSpec`, `Params` (implements `MultiVMWorkload`) | Most complex |
 
-The network workload is the most involved — it creates two VMs per configured count (a server and a client), needs a Kubernetes Service for DNS routing between them, and generates different cloud-init YAML for each role.
+The multi-VM workloads (network, tps) are the most involved — they create two VMs per configured count (a server and a client), need a Kubernetes Service for DNS routing between them, and generate different cloud-init YAML for each role.
+
+`MultiVMWorkload` extends `Workload` with `Roles()` and `UserdataForRole(role, namespace)`. The orchestrator type-asserts to this interface and dispatches per role; see [development.md](../development.md) for the implementation pattern.
 
 ```mermaid
 flowchart LR
@@ -263,15 +269,18 @@ If you want to dig deeper into how each component works:
 
 | I want to understand... | Look at... |
 |------------------------|------------|
-| How workloads define themselves | `internal/workloads/` — interface in `workload.go`, implementations in `cpu.go`, `memory.go`, etc. |
+| How workloads define themselves | `internal/workloads/` — `Workload` and `MultiVMWorkload` interfaces in `workload.go`; implementations in `cpu.go`, `memory.go`, `disk.go`, `database.go`, `network.go`, `tps.go`, `chaos_disk.go`, `chaos-network.go`, `chaos_process.go` |
+| The `diskSetupScript` helper for storage-backed workloads | `internal/workloads/workload.go` — generates the wait/format/mount/fstab script for `/dev/disk/by-id/virtio-<serial>` |
 | How VMs are built from workload data | `internal/vm/vm.go` — `BuildVMSpec()` and `CreateVM()` |
-| The CLI orchestration flow | `cmd/virtwork/main.go` — `runE()` and `cleanupE()` |
+| The CLI orchestration flow | `cmd/virtwork/main.go` — `runE()` and `cleanupE()`, including `namespaceDataVolumes` for per-VM DV naming |
 | Configuration loading | `internal/config/config.go` — `LoadConfig()` with Viper |
 | Cloud-init YAML generation | `internal/cloudinit/cloudinit.go` — `BuildCloudConfig()` |
 | Resource helpers (namespace, service, secret) | `internal/resources/resources.go` |
 | VM readiness polling | `internal/wait/wait.go` — concurrent VMI phase polling |
 | Cleanup by label selector | `internal/cleanup/cleanup.go` — `CleanupAll()` |
-| Audit logging | `internal/audit/` — `Auditor` interface, SQLite schema, records |
+| Audit logging | `internal/audit/` — `Auditor` interface, SQLite schema, records (see [audit-schema.md](../audit-schema.md) for the full schema) |
+| Structured logging | `internal/logging/logging.go` — `NewLogger(out, verbose)` returning a `*slog.Logger` |
 | Constants and defaults | `internal/constants/constants.go` |
+| Golden container disk image | `build/golden-image/` — Containerfile and build script for the optional pre-tooled image |
 
 For the full layered architecture with dependency diagrams, see [docs/architecture.md](../architecture.md).

From ecee4292de1336eb6075b457461fe7b656b4b2c7 Mon Sep 17 00:00:00 2001
From: Melvin Hillsman <mhillsma@redhat.com>
Date: Fri, 22 May 2026 10:28:09 -0500
Subject: [PATCH 08/17] docs(guide): fix full-suite VM count and add chaos/tps
 scenarios

Scenario 3 claimed the default run produces 8 VMs; with the three chaos
workloads now in DefaultRegistry, the actual count is 11. Update the VM
table to include chaos-disk, chaos-network, chaos-process, and add a
warning callout that chaos workloads need an isolated namespace.

Add two new scenarios:

- Scenario 8: deploy chaos-network and verify the tc/netem qdisc inside
  the VM with `tc qdisc show` and a ping.
- Scenario 9: deploy tps, watch the journalctl output for alternating
  TCP_RR and HTTP iterations, and show how to override file-size /
  iterations / duration via WorkloadConfig.Params in YAML.

Update the troubleshooting workload list to all nine names and add a row
for the chaos-network sch_netem kernel module dependency. Cross-link
deployment.md from the cluster-deployment scenario.

Signed-off-by: Melvin Hillsman <mhillsma@redhat.com>
---
 docs/guide/02-deploying-workloads.md | 81 ++++++++++++++++++++++++++--
 1 file changed, 76 insertions(+), 5 deletions(-)

diff --git a/docs/guide/02-deploying-workloads.md b/docs/guide/02-deploying-workloads.md
index 07566cb..c11eb7e 100644
--- a/docs/guide/02-deploying-workloads.md
+++ b/docs/guide/02-deploying-workloads.md
@@ -200,20 +200,25 @@ This creates 3 VMs, 3 cloud-init secrets, and 0 services:
 virtwork run
 ```
 
-With no `--workloads` flag, all workload types deploy. This creates **8 VMs** total — the network and tps workloads each create server/client pairs:
+With no `--workloads` flag, all nine workload types deploy. This creates **11 VMs** total — seven single-VM workloads plus the network and tps workloads, which each create a server/client pair:
 
 | VM Name | Workload | Role |
 |---------|----------|------|
 | `virtwork-cpu-0` | CPU | — |
-| `virtwork-database-0` | Database | — |
-| `virtwork-disk-0` | Disk | — |
 | `virtwork-memory-0` | Memory | — |
+| `virtwork-disk-0` | Disk | — |
+| `virtwork-database-0` | Database | — |
 | `virtwork-network-server-0` | Network | server |
 | `virtwork-network-client-0` | Network | client |
 | `virtwork-tps-server-0` | TPS | server |
 | `virtwork-tps-client-0` | TPS | client |
+| `virtwork-chaos-disk-0` | Chaos-disk | — |
+| `virtwork-chaos-network-0` | Chaos-network | — |
+| `virtwork-chaos-process-0` | Chaos-process | — |
+
+The network workload creates a `virtwork-iperf3-server` ClusterIP Service on port 5201. The tps workload creates a `virtwork-tps-server` ClusterIP Service on ports 12865 (netperf control), 12866 (netperf data), and 8080 (HTTP file transfer). The chaos workloads do not create Services — their behavior is confined to the VM they run in.
 
-The network workload creates a `virtwork-iperf3-server` ClusterIP Service on port 5201. The tps workload creates a `virtwork-tps-server` ClusterIP Service on ports 12865 (netperf control), 12866 (netperf data), and 8080 (HTTP file transfer).
+> ⚠️ The chaos workloads inject destructive behavior inside their VMs (disk fill, network latency/loss, random process kills). They are safe to run in an isolated namespace but should never share a namespace with workloads you care about. See [chaos-workloads.md](../chaos-workloads.md) for operational guidance.
 
 ### Scaling up
 
@@ -435,6 +440,71 @@ Set the `VIRTWORK_COMMAND` and `VIRTWORK_ARGS` environment variables in the Depl
 oc set env deploy/virtwork VIRTWORK_COMMAND=run VIRTWORK_ARGS="--workloads cpu,memory --vm-count 2"
 ```
 
+See [deployment.md](../deployment.md) for a manifest-by-manifest deep dive on the Kustomize layout.
+
+---
+
+## Scenario 8: Inject Network Chaos
+
+The chaos-network workload uses `tc` and `netem` to add latency and packet loss to the VM's egress interface — useful for testing how an application or monitoring stack reacts to a degraded network.
+
+```bash
+virtwork run --workloads chaos-network --ssh-user virtwork --ssh-key-file ~/.ssh/id_ed25519.pub
+```
+
+After the VM is ready, SSH in and confirm the qdisc is applied:
+
+```bash
+virtctl ssh --ssh-key ~/.ssh/id_ed25519 virtwork@virtwork-chaos-network-0 -n virtwork
+
+# Inside the VM:
+tc qdisc show dev eth0
+```
+
+You should see a `netem` qdisc with `delay 100ms` and `loss 5%` on the default-route interface. Verify with a ping to a known host:
+
+```bash
+ping -c 5 1.1.1.1
+```
+
+Expect ~100ms RTT and occasional packet loss in the output. The qdisc is managed by the `virtwork-chaos-network.service` systemd unit, so it will be reapplied on reboot.
+
+For chaos-disk and chaos-process, plus the full risk model, see [chaos-workloads.md](../chaos-workloads.md).
+
+---
+
+## Scenario 9: Transaction Performance with tps
+
+The tps workload measures transactions per second using two complementary tests on a server/client pair: `netperf TCP_RR` (small-packet round-trips at the transport layer) and an HTTP file-transfer loop (larger transfers at the application layer).
+
+```bash
+virtwork run --workloads tps --ssh-user virtwork --ssh-key-file ~/.ssh/id_ed25519.pub
+```
+
+Two VMs are created — `virtwork-tps-server-0` (runs `netserver` and a Python HTTP server) and `virtwork-tps-client-0` (loops through netperf and curl tests against the server's Service DNS name) — plus the `virtwork-tps-server` ClusterIP Service on ports 12865, 12866, and 8080.
+
+Watch the client's output:
+
+```bash
+virtctl ssh --ssh-key ~/.ssh/id_ed25519 virtwork@virtwork-tps-client-0 -n virtwork
+
+# Inside the VM:
+journalctl -u virtwork-tps.service -f
+```
+
+You will see alternating TCP_RR iteration blocks (transactions/sec) and HTTP iteration blocks (transfers/sec, MB/s).
+
+The defaults are 30 iterations × 60 seconds each, 10MB transfer file. To customize, set `workloads.tps.params` in a YAML config:
+
+```yaml
+workloads:
+  tps:
+    params:
+      file-size: "50M"
+      iterations: "10"
+      duration: "30"
+```
+
 ---
 
 ## Troubleshooting
@@ -446,4 +516,5 @@ oc set env deploy/virtwork VIRTWORK_COMMAND=run VIRTWORK_ARGS="--workloads cpu,m
 | VMs stuck in `Provisioning` | CDI not installed or no default StorageClass | Install the OpenShift Virtualization operator and ensure a default StorageClass exists |
 | Timeout waiting for readiness | Slow image pull or boot | Increase `--timeout` (default is 600s), or use `--no-wait` and check manually |
 | Cloud-init failed inside VM | Package install failure or script error | SSH in and check `/var/log/cloud-init-output.log` |
-| `unknown workload` error | Typo in `--workloads` | Available workloads: `cpu`, `database`, `disk`, `memory`, `network`, `tps` |
+| chaos-network: `tc` does nothing inside the VM | `sch_netem` kernel module not loaded | Confirm the VM image has `iproute-tc` and `kernel-modules-extra` available; check `journalctl -u virtwork-chaos-network.service` |
+| `unknown workload` error | Typo in `--workloads` | Available workloads: `cpu`, `memory`, `disk`, `database`, `network`, `tps`, `chaos-disk`, `chaos-network`, `chaos-process` |

From 519587d89db58b708fcb47ef51228f25793cb9cb Mon Sep 17 00:00:00 2001
From: Melvin Hillsman <mhillsma@redhat.com>
Date: Fri, 22 May 2026 10:29:31 -0500
Subject: [PATCH 09/17] docs(guide): extend 03-adding-a-workload with multi-VM
 and storage patterns

The tutorial's "Going Further" sections covered storage and multi-VM at
a sketch level. Two patterns that contributors actually need:

- For storage-backed workloads, always set the virtio Serial field on
  Disk and discover the device through /dev/disk/by-id/virtio-<serial>
  using the shared diskSetupScript helper. /dev/vdX ordering is not
  stable across reboots. Reference workloads: disk.go, database.go,
  chaos_disk.go.
- For multi-VM workloads, implement MultiVMWorkload with Roles() and
  UserdataForRole(); the orchestrator dispatches per role. Clients reach
  servers via in-cluster DNS (<service>.<namespace>.svc.cluster.local).
  Reference workloads: network.go (simple), tps.go (configurable Params).

Update the complexity spectrum diagram to show all nine workloads
grouped into the three tiers, and extend the checklist with explicit
multi-VM and storage-backed subsections.

Signed-off-by: Melvin Hillsman <mhillsma@redhat.com>
---
 docs/guide/03-adding-a-workload.md | 84 ++++++++++++++++++++++--------
 1 file changed, 62 insertions(+), 22 deletions(-)

diff --git a/docs/guide/03-adding-a-workload.md b/docs/guide/03-adding-a-workload.md
index 4bf853e..e7b8339 100644
--- a/docs/guide/03-adding-a-workload.md
+++ b/docs/guide/03-adding-a-workload.md
@@ -466,23 +466,30 @@ go run ./cmd/virtwork cleanup
 
 ## Going Further
 
-### Adding a Data Disk
+### Adding a Data Disk (Storage-Backed Workloads)
 
-If your workload needs persistent storage (for example, a workload that writes benchmark results to disk), override three methods. Look at `internal/workloads/disk.go` for the complete pattern:
+If your workload needs persistent storage (for example, a workload that writes benchmark results to disk), override three methods. Look at `internal/workloads/disk.go`, `database.go`, or `chaos_disk.go` for the complete pattern. The key things to get right:
 
 ```go
-// DataVolumeTemplates returns a CDI DataVolumeTemplateSpec.
+// DataVolumeTemplates returns a CDI DataVolumeTemplateSpec. The orchestrator
+// suffixes the template name with the VM name (namespaceDataVolumes in
+// cmd/virtwork/main.go) to avoid collisions when --vm-count > 1, so use the
+// un-suffixed base name here.
 func (w *MyWorkload) DataVolumeTemplates() []kubevirtv1.DataVolumeTemplateSpec {
     return []kubevirtv1.DataVolumeTemplateSpec{
         vm.BuildDataVolumeTemplate("my-data", w.DataDiskSize),
     }
 }
 
-// ExtraDisks adds the disk definition to the VM spec.
+// ExtraDisks adds the disk definition to the VM spec. ALWAYS set the Serial
+// field — the in-VM script discovers the device via
+// /dev/disk/by-id/virtio-<serial>, which is stable across reboots and
+// migrations (unlike /dev/vdX, which is not).
 func (w *MyWorkload) ExtraDisks() []kubevirtv1.Disk {
     return []kubevirtv1.Disk{
         {
-            Name: "datadisk",
+            Name:   "datadisk",
+            Serial: "virtwork-mydata",
             DiskDevice: kubevirtv1.DiskDevice{
                 Disk: &kubevirtv1.DiskTarget{Bus: "virtio"},
             },
@@ -490,7 +497,8 @@ func (w *MyWorkload) ExtraDisks() []kubevirtv1.Disk {
     }
 }
 
-// ExtraVolumes links the disk to the DataVolume.
+// ExtraVolumes links the disk to the DataVolume. The Name must match
+// ExtraDisks; the DataVolume.Name must match the template name above.
 func (w *MyWorkload) ExtraVolumes() []kubevirtv1.Volume {
     return []kubevirtv1.Volume{
         {
@@ -503,27 +511,46 @@ func (w *MyWorkload) ExtraVolumes() []kubevirtv1.Volume {
 }
 ```
 
-All three methods must return matching `Name` values — the disk, volume, and DataVolume are linked by name.
+In your cloud-init, write the shared `diskSetupScript(serial, mountPoint)` helper as the first script and run it from `runcmd` before the workload service starts. It waits for the `/dev/disk/by-id/virtio-<serial>` symlink, formats with XFS if empty, mounts at `mountPoint`, and writes `/etc/fstab` so the mount survives reboots.
+
+```go
+return w.BuildCloudConfig(CloudConfigOpts{
+    WriteFiles: []WriteFile{
+        {
+            Path:        "/usr/local/bin/virtwork-disk-setup.sh",
+            Content:     diskSetupScript("virtwork-mydata", "/mnt/data"),
+            Permissions: "0755",
+        },
+        // ... workload service unit and script ...
+    },
+    RunCmd: [][]string{
+        {"/usr/local/bin/virtwork-disk-setup.sh"},
+        {"systemctl", "daemon-reload"},
+        {"systemctl", "enable", "--now", "virtwork-my-workload.service"},
+    },
+})
+```
 
 ### Making It Multi-VM
 
-If you wanted nginx on one VM and `ab` on another (for more realistic cross-network metrics), you would implement the `MultiVMWorkload` interface. See `internal/workloads/network.go` for the complete pattern:
+If your workload needs more than one role of VM (a server and one or more clients, for example), implement the `MultiVMWorkload` interface. The two canonical references are `internal/workloads/network.go` (simplest — one Service port, iperf3) and `internal/workloads/tps.go` (multi-port Service with configurable `Params` for `file-size`, `iterations`, `duration`).
 
-1. Add a `Namespace` field to your struct (the client needs the server's DNS name)
-2. Implement `UserdataForRole(role, namespace) (string, error)` — return different cloud-init YAML for `"server"` vs `"client"`
-3. Override `VMCount()` to return `count * 2`
-4. Override `RequiresService()` to return `true`
-5. Implement `ServiceSpec()` to create a ClusterIP Service targeting the server VM by label selector
+1. Add a `Namespace` field to your struct — the client needs it to build the server's in-cluster DNS name.
+2. Implement `Roles() []string` (e.g., `[]string{"server", "client"}`).
+3. Implement `UserdataForRole(role, namespace) (string, error)` — return different cloud-init YAML per role. The orchestrator dispatches per role; the client constructs `<service>.<namespace>.svc.cluster.local` and never polls for pod IPs.
+4. Override `VMCount()` to return `count * len(Roles())` so the orchestrator creates the right number of VMs.
+5. Override `RequiresService()` to return `true`.
+6. Implement `ServiceSpec()` to create a ClusterIP Service. Its selector should match `virtwork/role: server` (and ideally `app.kubernetes.io/component: <your-workload>`) — the orchestrator applies the `virtwork/role` label to each VM automatically.
 
-The orchestrator detects `MultiVMWorkload` via type assertion and calls `UserdataForRole()` for each VM instead of `CloudInitUserdata()`.
+The orchestrator detects `MultiVMWorkload` via type assertion and calls `UserdataForRole()` per role/instance instead of `CloudInitUserdata()`.
 
 ### Workload Complexity Spectrum
 
 ```mermaid
 flowchart LR
-    A["<b>Simple</b><br/>CPU, Memory, HTTP<br/><i>Name + CloudInit only</i>"]
-    B["<b>With Storage</b><br/>Database, Disk<br/><i>+ DataVolumeTemplates<br/>+ ExtraDisks/Volumes</i>"]
-    C["<b>Multi-VM</b><br/>Network<br/><i>+ MultiVMWorkload<br/>+ Service + VMCount</i>"]
+    A["<b>Simple</b><br/>CPU, Memory, Chaos-process<br/><i>Name + CloudInit only</i>"]
+    B["<b>With Storage</b><br/>Disk, Database, Chaos-disk<br/><i>+ DataVolumeTemplates<br/>+ ExtraDisks (with Serial)<br/>+ ExtraVolumes<br/>+ diskSetupScript</i>"]
+    C["<b>Multi-VM</b><br/>Network, TPS<br/><i>+ MultiVMWorkload<br/>(Roles, UserdataForRole)<br/>+ Service + VMCount</i>"]
     A --> B --> C
 ```
 
@@ -534,17 +561,30 @@ Start simple. Add complexity only when the workload needs it.
 Before submitting a new workload, verify:
 
 - [ ] Workload struct embeds `BaseWorkload`
-- [ ] Constructor follows the `NewXWorkload(cfg, sshUser, sshPassword, sshKeys)` signature
-- [ ] `Name()` returns a lowercase, single-word identifier
+- [ ] Constructor follows the `NewXWorkload(cfg, sshUser, sshPassword, sshKeys)` signature (or extended signature for storage / namespace-aware workloads)
+- [ ] `Name()` returns a lowercase, hyphen-or-single-word identifier
 - [ ] `CloudInitUserdata()` calls `w.BuildCloudConfig()` (not `cloudinit.BuildCloudConfig()`)
-- [ ] Packages used are available in Fedora's default repos
+- [ ] Packages used are available in Fedora's default repos (or pre-installed in the golden image, if applicable)
 - [ ] Systemd unit has `Restart=always` and `WantedBy=multi-user.target`
 - [ ] Tests cover: Name, packages, systemd unit content, valid YAML, VMResources, defaults for optional methods
-- [ ] Registered in `DefaultRegistry()` with a factory function
-- [ ] Added to `AllWorkloadNames` (alphabetical order)
+- [ ] Registered in `DefaultRegistry()` (`internal/workloads/registry.go`) with a factory function
 - [ ] Existing registry/orchestration test counts updated
 - [ ] `go test ./...` passes
 - [ ] `go test -race ./...` passes
 - [ ] `--dry-run` produces a valid VM spec
 
+**If multi-VM:**
+
+- [ ] Implements `Roles() []string`
+- [ ] Implements `UserdataForRole(role, namespace) (string, error)`
+- [ ] `VMCount()` returns `Config.VMCount * len(Roles())`
+- [ ] `ServiceSpec().Spec.Selector` includes `virtwork/role: <server-role>` and `app.kubernetes.io/component: <name>`
+- [ ] Client userdata builds the server DNS as `<service>.<namespace>.svc.cluster.local`
+
+**If storage-backed:**
+
+- [ ] `DataVolumeTemplates()` returns templates with stable base names (orchestrator suffixes with VM name)
+- [ ] `ExtraDisks()` sets the `Serial` field on each Disk
+- [ ] Cloud-init runs `diskSetupScript(serial, mountPoint)` from `runcmd` before any service that uses the mount
+
 See [docs/development.md](../development.md) for the reference version of the "Adding a New Workload" checklist.

From 62c940ddd6711e5d6ea27d29217a3b271eace4f7 Mon Sep 17 00:00:00 2001
From: Melvin Hillsman <mhillsma@redhat.com>
Date: Fri, 22 May 2026 10:30:28 -0500
Subject: [PATCH 10/17] docs: mark implementation-plan and original design plan
 as historical
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Both documents describe the project at an earlier point in time:

- implementation-plan.md captures the original layered build through
  phase 12, before chaos workloads, tps, structured logging, and the
  golden image were added.
- openshift-virtualization-workload-automation.md was written before any
  application code existed and describes initial design rationale.

Neither should be updated piecemeal — they become inconsistent if treated
as living docs. Prepend a banner to each pointing at the current source
of truth (architecture.md, chaos-workloads.md, configuration.md, etc.)
and leave the body unchanged. Preserves the historical record while
making the freshness obvious to readers.

Signed-off-by: Melvin Hillsman <mhillsma@redhat.com>
---
 docs/implementation-plan.md                          | 12 ++++++++++++
 docs/openshift-virtualization-workload-automation.md | 10 ++++++++++
 2 files changed, 22 insertions(+)

diff --git a/docs/implementation-plan.md b/docs/implementation-plan.md
index ec8d4ba..be00b40 100644
--- a/docs/implementation-plan.md
+++ b/docs/implementation-plan.md
@@ -1,5 +1,17 @@
 # Virtwork Implementation Plan
 
+> **Historical snapshot — original phased build (phases 0–12).**
+>
+> This document describes the original incremental plan executed to build virtwork through phase 12. It is preserved as a record of the layered design and as context for why packages are organized the way they are. **Subsequent additions are not reflected here.** For the current state, see:
+>
+> - Chaos workloads (chaos-disk, chaos-network, chaos-process) → [chaos-workloads.md](chaos-workloads.md)
+> - TPS workload → [architecture.md](architecture.md) and [guide/02-deploying-workloads.md](guide/02-deploying-workloads.md)
+> - Structured logging (`internal/logging`) → [architecture.md](architecture.md) and [development.md](development.md)
+> - Golden container disk image → [../build/golden-image/README.md](../build/golden-image/README.md)
+> - Audit schema (5 tables) → [audit-schema.md](audit-schema.md)
+> - Complete configuration reference → [configuration.md](configuration.md)
+> - OpenShift deployment deep-dive → [deployment.md](deployment.md)
+
 This document breaks the [design plan](openshift-virtualization-workload-automation.md) into incremental phases. Each phase produces a testable, committable increment that builds on the previous one. See [architecture.md](architecture.md) for the high-level architecture and diagrams.
 
 ---
diff --git a/docs/openshift-virtualization-workload-automation.md b/docs/openshift-virtualization-workload-automation.md
index 0141049..3e757b3 100644
--- a/docs/openshift-virtualization-workload-automation.md
+++ b/docs/openshift-virtualization-workload-automation.md
@@ -1,5 +1,15 @@
 # Plan: OpenShift Virtualization Workload Automation ("virtwork")
 
+> **Historical snapshot — original design rationale.**
+>
+> This document captured the motivation and design choices at project inception, when no application code existed yet. It is preserved as context for *why* the layered architecture, the deploy-and-exit model, and the systemd-inside-VM workload lifecycle were chosen. **Implementation has since evolved.** For the current state, see:
+>
+> - Architecture and design decisions → [architecture.md](architecture.md)
+> - Built-in workloads (now nine total) → [README.md](../README.md#workloads), [architecture.md](architecture.md), [chaos-workloads.md](chaos-workloads.md)
+> - Developer guide and patterns → [development.md](development.md)
+> - Configuration reference → [configuration.md](configuration.md)
+> - OpenShift deployment → [deployment.md](deployment.md)
+
 ## Context
 
 This codebase (`virtwork`) is a Go CLI tool for creating VMs on an OpenShift cluster with OpenShift Virtualization (CNV) already installed. The goal is to run continuous, varied workloads (CPU, memory, database, network, disk I/O) inside those VMs so that monitoring tools (Prometheus, Grafana, etc.) have realistic metrics to observe. The codebase currently has a `go.mod` and documentation — no application code yet.

From 47eb09c8365bc336016a9ce770a5a5b6d7b25a78 Mon Sep 17 00:00:00 2001
From: Melvin Hillsman <mhillsma@redhat.com>
Date: Fri, 22 May 2026 10:31:56 -0500
Subject: [PATCH 11/17] docs(golden-image): clarify which workloads use each
 pre-installed tool

The pre-installed-tools list mentioned iproute-tc and iptables-nft as
"for future chaos workloads", but two of the three chaos workloads
already exist (chaos-network uses tc/netem; chaos-disk uses fallocate/dd
from the base image). Update the list to:

- Map each tool to the specific workload(s) that need it
- Add netperf + python3 for the tps workload (which were already in the
  Containerfile but unlisted)
- Add procps-ng for chaos-process
- Note that iptables-nft is still future-only (network partition tests)
- Mention that storage-backed workloads rely only on base-image utilities
  (blkid, mkfs.xfs, mount) via the diskSetupScript helper

Add a Related Docs section pointing back to the chaos-workloads guide,
deployment guide, and configuration reference.

Signed-off-by: Melvin Hillsman <mhillsma@redhat.com>
---
 build/golden-image/README.md | 19 +++++++++++++++----
 1 file changed, 15 insertions(+), 4 deletions(-)

diff --git a/build/golden-image/README.md b/build/golden-image/README.md
index 7161bea..7685fa3 100644
--- a/build/golden-image/README.md
+++ b/build/golden-image/README.md
@@ -11,11 +11,16 @@ The golden image is based on `quay.io/containerdisks/fedora:41` and includes all
 - **stress-ng** — CPU and memory stress testing (cpu, memory workloads)
 - **fio** — Flexible I/O tester for disk benchmarking (disk workload)
 - **iperf3** — Network performance testing (network workload)
+- **netperf** — TCP_RR transaction performance (tps workload)
+- **python3** — HTTP file server for the tps workload's application-layer transfers
 - **postgresql-server** — PostgreSQL database with pgbench (database workload)
-- **iproute-tc** — Traffic control for network chaos engineering (future chaos workloads)
-- **iptables-nft** — Firewall rules for network partition simulation (future chaos workloads)
+- **procps-ng** — `ps`, `pkill`, `kill` for chaos-process
+- **iproute-tc** — Traffic control (`tc`) and the `sch_netem` kernel module hooks used by the chaos-network workload to inject latency and packet loss
+- **iptables-nft** — Firewall rules; reserved for future network partition / blackhole scenarios
 
-Additional tools like `fallocate`, `dd`, `kill`, and `pkill` are already present in the base Fedora image.
+Additional tools like `fallocate` and `dd` (used by chaos-disk) are already present in the base Fedora image.
+
+Workloads that need persistent storage (disk, database, chaos-disk) discover their data volume through `/dev/disk/by-id/virtio-<serial>` using the shared `diskSetupScript` helper — they do not depend on any tools beyond the standard userspace utilities already in the base image (`blkid`, `mkfs.xfs`, `mount`, `readlink`).
 
 ## Building
 
@@ -131,11 +136,17 @@ The current approach is simpler and maintains backward compatibility.
 ## Future Enhancements
 
 1. **Multi-architecture support**: Build for arm64 in addition to amd64
-2. **Automated builds**: GitHub Actions workflow on schedule
+2. **Automated builds**: CI workflow on schedule
 3. **Image scanning**: Add Trivy security scanning
 4. **Semantic versioning**: Pin specific package versions for reproducibility
 5. **Image variants**: Create minimal/full variants for different use cases
 
+## Related Docs
+
+- [docs/chaos-workloads.md](../../docs/chaos-workloads.md) — operator guide for the chaos workloads that use `iproute-tc`, `procps-ng`, and `fallocate`
+- [docs/deployment.md](../../docs/deployment.md) — how to set the golden image as the default container disk in your deployment
+- [docs/configuration.md](../../docs/configuration.md) — `--container-disk-image` flag, `VIRTWORK_CONTAINER_DISK_IMAGE` env var, and `container_disk_image` YAML key
+
 ## License
 
 Apache License 2.0. See [LICENSE](../../LICENSE).

From c23133c9dc5299827a9179bdd3a059e8854df318 Mon Sep 17 00:00:00 2001
From: Melvin Hillsman <mhillsma@redhat.com>
Date: Fri, 22 May 2026 10:32:30 -0500
Subject: [PATCH 12/17] docs(claude): note logging policy, DCO trailer, doc
 conventions
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Add four guidance items that came up while overhauling documentation:

- DCO Signed-off-by is required by a commit-msg hook; do not add a
  Co-Authored-By trailer
- internal/logging.NewLogger() is the canonical logger for I/O code;
  the pure data layer (workloads, cloudinit) does not log
- docs/README.md is the index for the docs directory
- Mermaid for diagrams; no bare issue numbers in published docs
- The two historical snapshots (implementation-plan.md,
  openshift-virtualization-workload-automation.md) are intentionally
  frozen — new content goes into the live docs

Also clarify that engineering journals being absent is the policy, not
a missing directory.

Signed-off-by: Melvin Hillsman <mhillsma@redhat.com>
---
 CLAUDE.md | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/CLAUDE.md b/CLAUDE.md
index d17099a..fb5da43 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -6,6 +6,7 @@
 - Use conventional commit messages (feat:, fix:, test:, docs:, refactor:, chore:)
 - DO NOT commit secrets or other sensitive information
 - DO NOT commit engineering journals
+- Every commit must include a DCO `Signed-off-by` trailer (enforced by `commit-msg` hook); do NOT add a `Co-Authored-By` trailer
 
 ## Role Context
 - Expert Go developer
@@ -17,6 +18,7 @@
 - BDD: Use Ginkgo BDD framework
 - Use goroutines and channels for concurrency
 - Use mermaid for documentation diagrams
+- Use `internal/logging.NewLogger()` for structured logging in any code that performs I/O; do not use `fmt.Fprintf` or `log.Printf`. The pure data layer (`internal/workloads`, `internal/cloudinit`) should not log.
 
 ## Testing
 - Unit tests: `{source}_test.go` files alongside source
@@ -27,4 +29,11 @@
 ## Database
 - When a database is needed, use SQLite for local testing
 - Strictly adhere to PostgreSQL syntax standards (e.g., use standard SQL for dates, avoid loose typing)
-- Assume the production DB is strict.
\ No newline at end of file
+- Assume the production DB is strict.
+
+## Documentation Conventions
+- `docs/README.md` is the index/TOC for the `docs/` directory; link new docs from there.
+- Use mermaid for architecture, flow, sequence, ERD, and topology diagrams.
+- Do NOT reference bare issue numbers (`#123`) in published docs — if context from an issue matters, fold the relevant information into the doc prose so readers don't need to chase the tracker.
+- Two docs are intentionally frozen as historical snapshots and should not be updated piecemeal: `docs/implementation-plan.md` and `docs/openshift-virtualization-workload-automation.md`. Add new content to the live docs instead (architecture.md, development.md, configuration.md, etc.).
+- Engineering journals are not committed to this repo — that is the policy, not a missing directory.
\ No newline at end of file

From 19efb8786f19c7976e176e24cf95c8b544c0dab1 Mon Sep 17 00:00:00 2001
From: Melvin Hillsman <mhillsma@redhat.com>
Date: Fri, 22 May 2026 10:32:59 -0500
Subject: [PATCH 13/17] docs: add docs/README.md as documentation index
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

There are now twelve documents under docs/ plus the golden-image README
and the top-level README. Add a TOC so readers can navigate without
having to discover docs by guessing filenames or scrolling through
git history.

Group entries by purpose: Start Here (README + architecture), Guides
(narrative walkthroughs), Reference (config / chaos / audit-schema /
deployment / kube-burner comparison), Contributor (development guide
+ golden-image build), Historical (the two intentionally frozen
snapshots).

Note the directory-wide conventions at the bottom — mermaid for
diagrams, no bare issue numbers, historical docs stay frozen — so
future contributors don't reintroduce the patterns this overhaul
removed.

Signed-off-by: Melvin Hillsman <mhillsma@redhat.com>
---
 docs/README.md | 46 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)
 create mode 100644 docs/README.md

diff --git a/docs/README.md b/docs/README.md
new file mode 100644
index 0000000..934df8c
--- /dev/null
+++ b/docs/README.md
@@ -0,0 +1,46 @@
+# Virtwork Documentation
+
+This is the documentation index for virtwork. Start with the top-level [README](../README.md) for project orientation and quick start; come here when you need a specific reference.
+
+## Start Here
+
+- [../README.md](../README.md) — Project overview, motivation, installation, quick start, CLI reference, configuration summary, OpenShift deployment overview, architecture summary
+- [architecture.md](architecture.md) — Layered architecture, mermaid diagrams, concurrency model, workload class diagram, key design decisions
+
+## Guides
+
+Hands-on, narrative walkthroughs for first-time users and new contributors.
+
+- [guide/01-overview.md](guide/01-overview.md) — Mental model: the deploy-and-exit lifecycle, the `Workload` and `MultiVMWorkload` interfaces, where to find things in the code
+- [guide/02-deploying-workloads.md](guide/02-deploying-workloads.md) — Nine scenarios from dry-run to chaos to cluster-side deployment, with copy-pasteable commands and expected output
+- [guide/03-adding-a-workload.md](guide/03-adding-a-workload.md) — End-to-end TDD walkthrough that builds a new workload from scratch; covers simple, storage-backed, and multi-VM patterns
+
+## Reference
+
+Targeted references — read when you need a specific fact.
+
+- [configuration.md](configuration.md) — Complete reference for every CLI flag, environment variable, YAML key, and ConfigMap key (including per-workload parameters)
+- [chaos-workloads.md](chaos-workloads.md) — Operator guide for the three chaos engineering workloads (chaos-disk, chaos-network, chaos-process), including destructive-behavior warnings
+- [audit-schema.md](audit-schema.md) — SQLite audit database: five-table ERD, column-by-column reference, common queries, record lifecycle
+- [deployment.md](deployment.md) — OpenShift Kustomize deployment deep-dive: resource topology, RBAC scope, ConfigMap/Secret keys, image pinning, sizing, audit-DB persistence
+- [virtwork-vs-kube-burner.md](virtwork-vs-kube-burner.md) — Positioning compared to kube-burner; what each tool measures and where they complement each other
+
+## Contributor
+
+- [development.md](development.md) — Environment setup, building, running unit/integration/E2E tests, cluster prerequisites, Makefile targets, adding a new workload, SSH/audit configuration, testing patterns, commit conventions
+- [../build/golden-image/README.md](../build/golden-image/README.md) — Building and using the optional Fedora-based golden container disk image with pre-installed workload tools
+
+## Historical
+
+These documents capture earlier project state and are intentionally not updated. They are preserved as context.
+
+- [implementation-plan.md](implementation-plan.md) — Original phased build plan (phases 0–12), pre-chaos / pre-tps / pre-logging
+- [openshift-virtualization-workload-automation.md](openshift-virtualization-workload-automation.md) — Original design rationale, written before any application code existed
+
+---
+
+> **Conventions for this directory**
+>
+> - Diagrams are written in mermaid so they render natively on GitHub. Update them in place when the code they describe changes.
+> - Bare issue numbers (`#123`) are not used in published docs — any context that matters is folded into the prose.
+> - The historical snapshots above are frozen on purpose. New content goes into the live docs, not into them.

From 717bbd317828684cd56ccc22a3fb23db1bf1042b Mon Sep 17 00:00:00 2001
From: Melvin Hillsman <mhillsma@redhat.com>
Date: Fri, 22 May 2026 10:34:17 -0500
Subject: [PATCH 14/17] docs: add chaos-workloads.md operator guide

The three chaos engineering workloads (chaos-disk, chaos-network,
chaos-process) had no operator-facing documentation. Add a single guide
that covers all three, structured so an operator can land on it cold
and decide whether/how to deploy.

Contents:

- Quick comparison table and a prominent destructive-behavior warning
  (impact confined to the VM; namespace isolation is the safety boundary).
- Per-workload section with what it does, a mermaid flow diagram where
  helpful, configuration knobs (YAML params with defaults and effect),
  the in-VM dependencies (iproute-tc + sch_netem for chaos-network),
  end-to-end example with virtctl ssh commands, and what to observe
  in monitoring.
- Operational guidance on running chaos alongside non-chaos workloads
  (cross-VM impact only happens via application reach), cleanup, and
  audit visibility.
- Contributor section pointing at the source files and the shared
  patterns (BaseWorkload, BuildCloudConfig, diskSetupScript) so anyone
  adding a fourth chaos workload knows where to look.

Configuration knobs are documented at the user-facing layer
(WorkloadConfig.Params keys like fill-percent, latency-ms, signal) so
this guide serves as the spec for how those parameters should be wired
through. The plumbing matches what tps already does today.

Signed-off-by: Melvin Hillsman <mhillsma@redhat.com>
---
 docs/chaos-workloads.md | 281 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 281 insertions(+)
 create mode 100644 docs/chaos-workloads.md

diff --git a/docs/chaos-workloads.md b/docs/chaos-workloads.md
new file mode 100644
index 0000000..a0f8af7
--- /dev/null
+++ b/docs/chaos-workloads.md
@@ -0,0 +1,281 @@
+# Chaos Workloads
+
+Virtwork ships three chaos engineering workloads that inject failures inside a VM so that monitoring stacks, alerting rules, application resilience patterns, and recovery procedures can be exercised against realistic — not synthetic — disturbances.
+
+| Workload | What it does | Tool(s) | Storage |
+|---|---|---|---|
+| **chaos-disk** | Fills the data disk to a target percentage, sleeps, releases, repeats | `fallocate`, `dd` | DataVolume mounted at `/mnt/data` |
+| **chaos-network** | Adds latency and packet loss on the VM's egress interface | `tc` + `netem` | None |
+| **chaos-process** | Randomly signals non-essential processes inside the VM | shell + `ps`/`kill` | None |
+
+All three run as systemd services (`virtwork-chaos-disk.service`, `virtwork-chaos-network.service`, `virtwork-chaos-process.service`) and restart automatically on failure or VM reboot.
+
+---
+
+## ⚠️ Destructive-Behavior Warning
+
+Chaos workloads are **deliberately destructive within their VM**:
+
+- **chaos-disk** consumes nearly all of the data PVC and releases it on a loop. While it's in the "fill" phase, any other process on the VM that writes to `/mnt/data` will see `ENOSPC`.
+- **chaos-network** rewrites the egress qdisc on the default-route interface. All outbound traffic from that VM — including health checks, metric scrapes, and SSH responses — incurs the configured latency and drop rate.
+- **chaos-process** sends signals (default `SIGTERM`) to randomly selected processes every 30 seconds. Long-running tools that are not in the exclusion list can be killed at any time.
+
+The destructive effects are **confined to the VM** the workload runs in. Virtwork does not have a platform-level kill switch. Use namespace isolation as your safety boundary: deploy chaos workloads in their own namespace, or at least never alongside workloads you care about.
+
+```bash
+# RECOMMENDED: dedicated chaos namespace
+virtwork run --namespace virtwork-chaos --workloads chaos-disk,chaos-network,chaos-process
+
+# DO NOT: mix with workloads you intend to observe
+virtwork run --workloads cpu,database,chaos-disk    # database and chaos-disk share the namespace
+```
+
+Cleanup is unchanged from any other workload — `virtwork cleanup` deletes the VMs and the chaos behavior stops with them.
+
+---
+
+## chaos-disk
+
+### What it does
+
+Continuously fills the mounted data disk to a configurable percentage, holds the fill for a configurable duration, then releases it. This produces sustained disk-pressure events that exercise:
+
+- Filesystem-fill alerts in your monitoring stack
+- Eviction or pause behavior in components that watch disk usage
+- Recovery paths that depend on freeing disk space
+
+### Flow
+
+```mermaid
+flowchart LR
+    A[Compute target_kb<br/>= total * fill-percent] --> B{fill_kb > 0?}
+    B -->|yes| C[fallocate or dd<br/>chaos-disk-fill file]
+    B -->|no| D[Skip fill]
+    C --> E[Sleep fill-sleep<br/>default 60s]
+    D --> E
+    E --> F[rm chaos-disk-fill]
+    F --> G[Sleep release-sleep<br/>default 30s]
+    G --> A
+```
+
+### Configuration
+
+| YAML key (`workloads.chaos-disk.params.*`) | Default | Effect |
+|---|---|---|
+| `mount` | `/mnt/data` | Mountpoint of the data disk to fill |
+| `fill-percent` | `90` | Target fill percentage of the mountpoint |
+| `fill-sleep` | `60` | Seconds to hold the fill before releasing |
+| `release-sleep` | `30` | Seconds to wait empty before refilling |
+
+Example:
+
+```yaml
+workloads:
+  chaos-disk:
+    enabled: true
+    vm_count: 1
+    cpu_cores: 1
+    memory: 1Gi
+    params:
+      fill-percent: "80"
+      fill-sleep: "120"
+      release-sleep: "60"
+```
+
+The data disk size comes from `--disk-size` / `VIRTWORK_DATA_DISK_SIZE` / `data_disk_size` (default `10Gi`). The disk is provisioned as a CDI `DataVolume`, mounted in-VM via the virtio Serial pattern (`/dev/disk/by-id/virtio-virtwork-chdisk`), formatted XFS, and persisted in `/etc/fstab` by the shared `diskSetupScript` helper.
+
+### Example
+
+```bash
+virtwork run --workloads chaos-disk --disk-size 5Gi --ssh-user virtwork --ssh-key-file ~/.ssh/id_ed25519.pub
+
+# Watch the fill/release cycle from inside the VM:
+virtctl ssh --ssh-key ~/.ssh/id_ed25519 virtwork@virtwork-chaos-disk-0 -n virtwork
+watch -n 5 'df -h /mnt/data'
+```
+
+You should see `/mnt/data` use rise to ~90%, hold for ~60s, drop back near baseline, hold for ~30s, and repeat.
+
+### What to observe
+
+- `kubelet_volume_stats_used_bytes{persistentvolumeclaim=~"virtwork-chaos-disk.*"}` from kube-state-metrics
+- Pod or VMI events around disk pressure
+- Any application metrics tied to free-space thresholds you have configured
+
+---
+
+## chaos-network
+
+### What it does
+
+Applies a `tc` qdisc with `netem` to the VM's default-route interface, adding latency and packet loss to every egress packet. The qdisc is installed once on service start and removed cleanly on service stop, so a `systemctl restart virtwork-chaos-network` from inside the VM disables and re-enables the impairment.
+
+### Configuration
+
+| YAML key (`workloads.chaos-network.params.*`) | Default | Effect |
+|---|---|---|
+| `latency-ms` | `100` | One-way latency to inject (`netem delay <ms>ms`) |
+| `packet-loss-percent` | `5.0` | Packet drop rate (`netem loss <pct>%`) |
+
+Example:
+
+```yaml
+workloads:
+  chaos-network:
+    enabled: true
+    vm_count: 1
+    cpu_cores: 1
+    memory: 1Gi
+    params:
+      latency-ms: "250"
+      packet-loss-percent: "10"
+```
+
+### Dependencies
+
+The workload installs `iproute-tc` and loads the `sch_netem` kernel module on start. The base Fedora container disk has both available; the golden image (`build/golden-image/`) pre-installs `iproute-tc` to avoid the first-boot package install. `sch_netem` is loaded from the `kernel-modules-extra` package, which the cloud-init runs `dnf install` for on first boot.
+
+### Example
+
+```bash
+virtwork run --workloads chaos-network --ssh-user virtwork --ssh-key-file ~/.ssh/id_ed25519.pub
+
+# Inside the VM, confirm the qdisc is active:
+virtctl ssh --ssh-key ~/.ssh/id_ed25519 virtwork@virtwork-chaos-network-0 -n virtwork
+tc qdisc show dev eth0
+# qdisc netem 8001: root refcnt 2 limit 1000 delay 100ms loss 5%
+
+ping -c 10 1.1.1.1
+# Expect ~100ms RTT and occasional packet loss
+```
+
+### What to observe
+
+- Increased latency in your service-to-service traces (when chaos-network sits on the client side)
+- Retry / circuit-breaker behavior in HTTP clients
+- TCP retransmit counters (`netstat -s` inside the VM, or node-level metrics if you scrape them)
+
+---
+
+## chaos-process
+
+### What it does
+
+Every 30 seconds (configurable), randomly selects one process from the VM that is **not** in the exclusion list and sends it `SIGTERM` (configurable). Processes are eligible when their PID is at or above the configured minimum (default 1000) — this is a coarse filter to leave kernel threads and core systemd-managed processes alone.
+
+### Exclusion list
+
+The following process-name patterns are never targeted (built into the script):
+
+- `systemd`
+- `sshd`
+- `dbus`
+- `agetty`
+- `auditd`
+- `rsyslogd`
+- `chronyd`
+- `NetworkManager`
+- `bash`, `sh`
+- `cloud-init`
+- `virtwork-*` (the chaos workload's own service)
+
+### Configuration
+
+| YAML key (`workloads.chaos-process.params.*`) | Default | Effect |
+|---|---|---|
+| `signal` | `SIGTERM` | Signal sent to the selected process (`SIGTERM`, `SIGKILL`, `SIGINT`, …) |
+| `interval` | `30` | Seconds between kill attempts |
+| `min-pid` | `1000` | Minimum PID considered eligible |
+
+Example:
+
+```yaml
+workloads:
+  chaos-process:
+    enabled: true
+    vm_count: 1
+    cpu_cores: 1
+    memory: 512Mi
+    params:
+      signal: "SIGKILL"
+      interval: "15"
+      min-pid: "500"
+```
+
+### Example
+
+```bash
+virtwork run --workloads chaos-process --ssh-user virtwork --ssh-key-file ~/.ssh/id_ed25519.pub
+
+# Watch the chaos log from inside the VM:
+virtctl ssh --ssh-key ~/.ssh/id_ed25519 virtwork@virtwork-chaos-process-0 -n virtwork
+journalctl -u virtwork-chaos-process.service -f
+```
+
+Sample output:
+
+```
+[2026-05-22 14:00:00] Starting chaos-process workload
+[2026-05-22 14:00:00] Configuration: SIGNAL=SIGTERM INTERVAL=30s MIN_PID=1000
+[2026-05-22 14:00:00] Sending SIGTERM to PID 1437: 1437 cron /usr/sbin/cron -n
+[2026-05-22 14:00:00] Successfully sent SIGTERM to PID 1437
+[2026-05-22 14:00:30] Sending SIGTERM to PID 1502: 1502 anacron /usr/sbin/anacron -d -q -s
+```
+
+### What to observe
+
+- Process restarts driven by systemd `Restart=` directives in any services you install alongside
+- Alerts that fire when key processes disappear
+- Recovery time from kill to restart
+
+---
+
+## Operational Guidance
+
+### Running chaos alongside non-chaos workloads
+
+If you must run chaos workloads in the same namespace as non-chaos workloads (for example, to apply chaos to a test database), be aware that:
+
+- **chaos-disk** affects its own VM's data disk only — it cannot impact another VM's disk because each VM gets its own DataVolume.
+- **chaos-network** affects egress from its own VM only — it does not affect traffic to or from sibling VMs.
+- **chaos-process** affects processes inside its own VM only.
+
+Cross-VM impact happens only when the *application* under test reaches across VMs and is itself affected by the chaos (e.g., a client VM with chaos-network installed will see degraded performance talking to any server VM).
+
+### Cleanup
+
+```bash
+# Drop everything chaos-related
+virtwork cleanup --namespace virtwork-chaos --delete-namespace
+
+# Or by run ID
+virtwork cleanup --run-id <uuid>
+```
+
+Cleanup is the same as any other virtwork resource — label selectors find chaos VMs and their data volumes are reclaimed automatically when the VMs are deleted (subject to the StorageClass reclaim policy).
+
+### Audit visibility
+
+Chaos workloads are recorded in the audit database exactly like any other workload. The `workload_details` row will have `workload_type` set to `chaos-disk` / `chaos-network` / `chaos-process`, and `events` will show `vm_created`, `vm_ready`, and on cleanup `vm_deleted` rows. See [audit-schema.md](audit-schema.md).
+
+---
+
+## For Contributors
+
+Source files:
+
+- `internal/workloads/chaos_disk.go` — `ChaosDiskWorkload` struct, `chaosDiskScript`, `chaosDiskSystemdUnit`
+- `internal/workloads/chaos-network.go` — `ChaosNetworkWorkload` struct, `chaosNetworkStartScript`, `chaosNetworkStopScript`, `chaosNetworkSystemdUnit`
+- `internal/workloads/chaos_process.go` — `ChaosProcessWorkload` struct, `chaosProcessScript`, `chaosProcessSystemdUnit`
+- `internal/workloads/registry.go` — `DefaultRegistry()` registers all three under the names `chaos-disk`, `chaos-network`, `chaos-process`
+
+All three embed `BaseWorkload` and use the standard `BuildCloudConfig(opts)` helper for SSH credential injection. chaos-disk additionally uses the shared `diskSetupScript(serial, mountPoint)` helper from `internal/workloads/workload.go` for the standard wait/format/mount/fstab pattern.
+
+The configuration knobs listed above are sourced from `WorkloadConfig.Params` — the same plumbing the tps workload uses for `file-size`, `iterations`, `duration`. See [development.md](development.md) for the multi-VM / storage-backed / configurable-params patterns.
+
+## Related Docs
+
+- [configuration.md](configuration.md) — full reference for every configuration knob, including chaos parameters
+- [audit-schema.md](audit-schema.md) — where chaos-workload events land in the audit database
+- [guide/02-deploying-workloads.md](guide/02-deploying-workloads.md) — Scenario 8 walks through chaos-network end-to-end
+- [development.md](development.md) — adding a new workload (including a new chaos workload)

From c3590ed6ab6715cc236ceea76c8e67cb4ced1bd0 Mon Sep 17 00:00:00 2001
From: Melvin Hillsman <mhillsma@redhat.com>
Date: Fri, 22 May 2026 10:35:47 -0500
Subject: [PATCH 15/17] docs: add audit-schema.md as the SQLite audit reference
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

audit_log was partially documented in the README's audit section but the
other four tables (workload_details, vm_details, resource_details,
events) had no documentation. Anyone querying the database had to
reverse-engineer the schema from `.schema` output.

Cover the schema end-to-end:

- A mermaid erDiagram showing all five tables, their PK/FK relationships,
  and the most useful columns at a glance
- One subsection per table with every column (type, NOT NULL, FK target,
  description) — sourced from internal/audit/schema.go
- Enumeration of the event_type values currently emitted by the
  orchestrator, with when each fires
- A sequenceDiagram showing the lifecycle: StartExecution → per-workload
  → per-service → per-VM → readiness → CompleteExecution
- Seven common queries covering the most useful operational questions
  (recent runs, VMs per run, events timeline, failed runs, cleanup
  linking, totals, average ready latency)
- Contributor section explaining how to add a column or event type, with
  the PostgreSQL-compatibility constraints (TEXT for ISO 8601 timestamps
  and JSON-shaped data) called out

Cross-link from configuration.md, deployment.md, and development.md.

Signed-off-by: Melvin Hillsman <mhillsma@redhat.com>
---
 docs/audit-schema.md | 356 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 356 insertions(+)
 create mode 100644 docs/audit-schema.md

diff --git a/docs/audit-schema.md b/docs/audit-schema.md
new file mode 100644
index 0000000..4545c9f
--- /dev/null
+++ b/docs/audit-schema.md
@@ -0,0 +1,356 @@
+# Audit Database Schema
+
+Every `virtwork run` and `virtwork cleanup` execution is recorded in a local SQLite database (`virtwork.db` by default) so that what happened, when, with what configuration, and with what outcome is always recoverable — even when the cluster itself is gone.
+
+This document is the reference for the schema: tables, columns, indexes, relationships, and how rows are produced over an execution's lifetime.
+
+## Storage and Implementation
+
+- **Backing store:** SQLite, opened in WAL mode for concurrent-read safety. The file path is `virtwork.db` by default, overridden via `--audit-db` / `VIRTWORK_AUDIT_DB` / `audit_db` YAML key. When deployed in-cluster the path is `/data/virtwork.db` on the audit PVC.
+- **Interface:** `internal/audit.Auditor` with two implementations: `SQLiteAuditor` (the real one) and `NoOpAuditor` (returned when `--no-audit` or `VIRTWORK_AUDIT=false` is set).
+- **PostgreSQL compatibility:** all `TEXT` timestamps use ISO 8601 strings; `linked_run_ids` is a JSON array stored as `TEXT` so the same shape migrates cleanly to PostgreSQL JSONB if needed later.
+- **DDL location:** `internal/audit/schema.go`. Record structs that map to inserts live in `internal/audit/records.go`.
+
+## Entity-Relationship Diagram
+
+```mermaid
+erDiagram
+    audit_log ||--o{ workload_details : "one run has many workloads"
+    audit_log ||--o{ vm_details       : "one run has many VMs"
+    audit_log ||--o{ resource_details : "one run has many resources"
+    audit_log ||--o{ events           : "one run has many events"
+    workload_details ||--o{ vm_details : "one workload has many VMs"
+    workload_details ||--o{ events     : "events optionally reference workload"
+    vm_details       ||--o{ events     : "events optionally reference VM"
+
+    audit_log {
+        INTEGER id PK
+        TEXT    run_id "UNIQUE — UUID for the execution"
+        TEXT    linked_run_ids "JSON array, set on cleanup"
+        TEXT    command "run | dry-run | cleanup"
+        TEXT    status "in_progress | success | failed"
+        TEXT    namespace
+        TEXT    workloads_csv
+        INTEGER total_vm_count
+        INTEGER dry_run
+        INTEGER ssh_auth_configured
+        TEXT    started_at
+        TEXT    completed_at
+        TEXT    error_summary
+    }
+
+    workload_details {
+        INTEGER id PK
+        INTEGER audit_id FK
+        TEXT    workload_type "cpu, tps, chaos-disk, ..."
+        INTEGER vm_count
+        INTEGER cpu_cores
+        TEXT    memory
+        INTEGER has_data_disk
+        INTEGER requires_service
+        TEXT    status "pending | created | failed"
+    }
+
+    vm_details {
+        INTEGER id PK
+        INTEGER audit_id FK
+        INTEGER workload_id FK
+        TEXT    vm_name
+        TEXT    component
+        TEXT    role "server | client | (empty)"
+        TEXT    phase
+        TEXT    status "planned | created | ready | deleted | failed"
+        TEXT    created_at
+        TEXT    ready_at
+        TEXT    deleted_at
+    }
+
+    resource_details {
+        INTEGER id PK
+        INTEGER audit_id FK
+        TEXT    resource_type "Service | Secret"
+        TEXT    resource_name
+        TEXT    status "created | deleted"
+    }
+
+    events {
+        INTEGER id PK
+        INTEGER audit_id FK
+        INTEGER vm_id FK
+        INTEGER workload_id FK
+        TEXT    event_type
+        TEXT    message
+        TEXT    error_detail
+        TEXT    occurred_at
+    }
+```
+
+## Tables
+
+### `audit_log` — one row per execution
+
+| Column | Type | Notes |
+|---|---|---|
+| `id` | INTEGER PK | Autoincrement |
+| `run_id` | TEXT UNIQUE NOT NULL | UUID applied to all K8s resources via the `virtwork/run-id` label |
+| `linked_run_ids` | TEXT | JSON array of run-IDs discovered during cleanup (NULL outside of cleanup) |
+| `command` | TEXT NOT NULL | `run`, `dry-run`, `cleanup`, or `cleanup --dry-run` |
+| `status` | TEXT NOT NULL | `in_progress` initially, `success` or `failed` on completion |
+| `kubeconfig_path` | TEXT | Path used to connect (NULL when in-cluster) |
+| `cluster_context` | TEXT | Reserved — currently unused |
+| `namespace` | TEXT NOT NULL | Target namespace |
+| `container_disk_image` | TEXT | Configured boot image |
+| `default_cpu_cores` | INTEGER | Global `--cpu-cores` default |
+| `default_memory` | TEXT | Global `--memory` default |
+| `data_disk_size` | TEXT | Global `--disk-size` default |
+| `workloads_csv` | TEXT | Comma-separated list of workload names requested |
+| `total_vm_count` | INTEGER | Total VMs planned across all workloads |
+| `total_workload_count` | INTEGER | Number of workload types deployed |
+| `dry_run` | INTEGER NOT NULL | 0 or 1 |
+| `ssh_auth_configured` | INTEGER NOT NULL | 1 if any SSH credential was provided. **Credentials are never stored.** |
+| `cleanup_mode` | TEXT | Reserved — currently unused |
+| `wait_for_ready` | INTEGER NOT NULL | 0 or 1; reflects `--no-wait` inverted |
+| `ready_timeout_seconds` | INTEGER | Effective readiness timeout |
+| `vms_deleted` | INTEGER | Cleanup only: count from `CleanupResult` |
+| `services_deleted` | INTEGER | Cleanup only |
+| `secrets_deleted` | INTEGER | Cleanup only |
+| `namespace_deleted` | INTEGER | Cleanup only: 0 or 1 |
+| `started_at` | TEXT NOT NULL | ISO 8601 timestamp |
+| `completed_at` | TEXT | NULL while in flight |
+| `error_summary` | TEXT | Populated when `status = 'failed'` |
+
+Indexes: `started_at`, `namespace`, `status`, `run_id`.
+
+### `workload_details` — one row per workload-type per execution
+
+| Column | Type | Notes |
+|---|---|---|
+| `id` | INTEGER PK | |
+| `audit_id` | INTEGER NOT NULL | FK → `audit_log.id` |
+| `workload_type` | TEXT NOT NULL | `cpu`, `memory`, `disk`, `database`, `network`, `tps`, `chaos-disk`, `chaos-network`, `chaos-process` |
+| `enabled` | INTEGER NOT NULL | 0 or 1 (always 1 in current orchestration) |
+| `vm_count` | INTEGER NOT NULL | Reported by `Workload.VMCount()` — `N` for single-VM, `N × len(Roles())` for multi-VM |
+| `cpu_cores` | INTEGER NOT NULL | Effective per-VM CPU cores |
+| `memory` | TEXT NOT NULL | Effective per-VM memory (e.g., `2Gi`) |
+| `has_data_disk` | INTEGER NOT NULL | 1 when `DataVolumeTemplates()` is non-empty |
+| `data_disk_size` | TEXT | NULL when `has_data_disk = 0` |
+| `requires_service` | INTEGER NOT NULL | 1 for multi-VM workloads (network, tps) |
+| `status` | TEXT NOT NULL | `pending` → `created` (after VM create succeeds) → `failed` |
+| `started_at` | TEXT | Reserved — currently unset |
+| `completed_at` | TEXT | Reserved — currently unset |
+
+Indexes: `audit_id`, `workload_type`.
+
+### `vm_details` — one row per VM
+
+| Column | Type | Notes |
+|---|---|---|
+| `id` | INTEGER PK | |
+| `audit_id` | INTEGER NOT NULL | FK → `audit_log.id` |
+| `workload_id` | INTEGER | FK → `workload_details.id` |
+| `vm_name` | TEXT NOT NULL | e.g., `virtwork-cpu-0`, `virtwork-network-server-0` |
+| `namespace` | TEXT NOT NULL | |
+| `component` | TEXT NOT NULL | Workload name (matches `workload_details.workload_type`) |
+| `role` | TEXT | `server` / `client` for multi-VM workloads; empty for single-VM |
+| `cpu_cores` | INTEGER NOT NULL | |
+| `memory` | TEXT NOT NULL | |
+| `container_disk_image` | TEXT NOT NULL | |
+| `has_data_disk` | INTEGER NOT NULL | |
+| `data_disk_size` | TEXT | |
+| `phase` | TEXT | Latest known VMI phase (`Pending`, `Scheduled`, `Running`, …) |
+| `status` | TEXT NOT NULL | `planned` → `created` → `ready` / `failed` / `deleted` |
+| `created_at` | TEXT | Set when `CreateVM` succeeds |
+| `ready_at` | TEXT | Set when readiness polling succeeds |
+| `deleted_at` | TEXT | Set during cleanup |
+
+Indexes: `audit_id`, `workload_id`, `vm_name`.
+
+### `resource_details` — one row per non-VM K8s resource
+
+Tracks Services and Secrets created by orchestration. (VMs go in `vm_details`.)
+
+| Column | Type | Notes |
+|---|---|---|
+| `id` | INTEGER PK | |
+| `audit_id` | INTEGER NOT NULL | FK → `audit_log.id` |
+| `resource_type` | TEXT NOT NULL | `Service` or `Secret` |
+| `resource_name` | TEXT NOT NULL | e.g., `virtwork-iperf3-server`, `virtwork-cpu-0-cloudinit` |
+| `namespace` | TEXT NOT NULL | |
+| `status` | TEXT NOT NULL | `created` or `deleted` |
+| `created_at` | TEXT | |
+| `deleted_at` | TEXT | Set during cleanup |
+
+Indexes: `audit_id`, `resource_type`.
+
+### `events` — append-only event log
+
+| Column | Type | Notes |
+|---|---|---|
+| `id` | INTEGER PK | |
+| `audit_id` | INTEGER NOT NULL | FK → `audit_log.id` |
+| `vm_id` | INTEGER | FK → `vm_details.id` (optional) |
+| `workload_id` | INTEGER | FK → `workload_details.id` (optional) |
+| `event_type` | TEXT NOT NULL | See enumeration below |
+| `message` | TEXT | Free-form |
+| `error_detail` | TEXT | Set when the event represents a failure |
+| `occurred_at` | TEXT NOT NULL | |
+
+Indexes: `audit_id`, `event_type`, `occurred_at`.
+
+**Common `event_type` values emitted by the orchestrator today:**
+
+| `event_type` | When emitted |
+|---|---|
+| `execution_started` | Two per run — once for "Starting" and once for "Planned N VMs across M workloads" |
+| `service_created` | After each Service is created |
+| `vm_created` | After each VM is successfully created on the cluster |
+| `vm_failed` | When `CreateVM` returns an error |
+| `vm_ready` | After readiness polling confirms `Running` |
+| `vm_timeout` | When a VM fails readiness check |
+| `cleanup_started` | Beginning of `cleanup` execution |
+| `cleanup_completed` | End of `cleanup` execution, with the deletion counts |
+
+---
+
+## Lifecycle: How Rows Appear
+
+```mermaid
+sequenceDiagram
+    participant CLI as cmd/virtwork
+    participant A as Auditor
+    participant DB as SQLite
+
+    CLI->>A: StartExecution(ctx, "run", cfg)
+    A->>DB: INSERT audit_log (run_id=UUID, status=in_progress, started_at=now)
+    A-->>CLI: (execID, runID)
+
+    loop per workload
+        CLI->>A: RecordWorkload(execID, WorkloadRecord)
+        A->>DB: INSERT workload_details (audit_id=execID, ...)
+        A-->>CLI: workloadID
+    end
+
+    loop per Service to create
+        CLI->>A: RecordResource(execID, ResourceRecord)
+        A->>DB: INSERT resource_details
+        CLI->>A: RecordEvent(execID, event_type=service_created)
+        A->>DB: INSERT events
+    end
+
+    loop per VM to create (errgroup)
+        CLI->>A: RecordVM(execID, workloadID, VMRecord)
+        A->>DB: INSERT vm_details (status=planned)
+        CLI->>A: RecordEvent(execID, event_type=vm_created)
+        A->>DB: INSERT events
+    end
+
+    Note over CLI: WaitForAllVMsReady ...
+
+    loop per VM ready/timeout
+        CLI->>A: RecordEvent(execID, event_type=vm_ready|vm_timeout)
+        A->>DB: INSERT events
+    end
+
+    CLI->>A: UpdateWorkloadStatus(workloadID, "created")
+    A->>DB: UPDATE workload_details SET status='created'
+
+    CLI->>A: CompleteExecution(execID, "success", "")
+    A->>DB: UPDATE audit_log SET status='success', completed_at=now
+```
+
+Cleanup is similar: a new `audit_log` row is created with `command='cleanup'`, resources are discovered by label, deleted, counted, and the `linked_run_ids` column is filled with the unique `virtwork/run-id` values that were present on the deleted resources.
+
+---
+
+## Common Queries
+
+### Recent executions
+
+```sql
+SELECT id, run_id, command, status, namespace, total_vm_count, started_at
+FROM audit_log
+ORDER BY id DESC
+LIMIT 10;
+```
+
+### VMs from a specific run
+
+```sql
+SELECT vm_name, component, role, cpu_cores, memory, status, ready_at
+FROM vm_details
+WHERE audit_id = (SELECT id FROM audit_log WHERE run_id = '<uuid>');
+```
+
+### Events timeline for a run
+
+```sql
+SELECT occurred_at, event_type, message
+FROM events
+WHERE audit_id = (SELECT id FROM audit_log WHERE run_id = '<uuid>')
+ORDER BY occurred_at;
+```
+
+### Failed runs in the last day
+
+```sql
+SELECT id, run_id, command, error_summary, started_at, completed_at
+FROM audit_log
+WHERE status = 'failed' AND started_at >= datetime('now', '-1 day')
+ORDER BY started_at DESC;
+```
+
+### Which cleanups touched a given run-id?
+
+```sql
+SELECT id, run_id, started_at, vms_deleted, services_deleted, secrets_deleted, linked_run_ids
+FROM audit_log
+WHERE command LIKE 'cleanup%' AND linked_run_ids LIKE '%<uuid>%';
+```
+
+### How many VMs per workload across all runs?
+
+```sql
+SELECT workload_type, SUM(vm_count) AS total_vms, COUNT(*) AS runs
+FROM workload_details
+GROUP BY workload_type
+ORDER BY total_vms DESC;
+```
+
+### Average VM-ready latency per workload type
+
+```sql
+SELECT
+  component,
+  AVG((julianday(ready_at) - julianday(created_at)) * 86400) AS avg_ready_seconds
+FROM vm_details
+WHERE ready_at IS NOT NULL AND created_at IS NOT NULL
+GROUP BY component
+ORDER BY avg_ready_seconds DESC;
+```
+
+---
+
+## For Contributors
+
+- DDL: `internal/audit/schema.go` (`schemaSQL` constant, executed once on `NewSQLiteAuditor`).
+- Insert/update methods: `internal/audit/audit.go` (`SQLiteAuditor` methods).
+- Record structs that map to inserts: `internal/audit/records.go` (`WorkloadRecord`, `VMRecord`, `ResourceRecord`, `EventRecord`).
+- `NoOpAuditor`: returned when audit is disabled. Implements the same interface with empty methods so call sites never need to nil-check.
+
+### Adding a new column
+
+1. Add the column to `schemaSQL` in `schema.go`. SQLite's `CREATE TABLE IF NOT EXISTS` will not alter an existing table — for production-like upgrades, add an explicit `ALTER TABLE` migration.
+2. Add the field to the relevant `Record` struct in `records.go`.
+3. Update the insert/update method in `audit.go` to include the new column.
+4. Update the relevant table section in this document.
+5. PostgreSQL compatibility: keep timestamps as ISO 8601 `TEXT`; use `TEXT` for JSON-shaped data (it maps cleanly to JSONB).
+
+### Adding a new event type
+
+Just call `RecordEvent(ctx, execID, EventRecord{EventType: "your_new_type", Message: "..."})` from the orchestrator. Add a row to the "Common `event_type` values" table above so the catalog stays current.
+
+## Related Docs
+
+- [configuration.md](configuration.md) — audit-related flags, env vars, YAML keys
+- [deployment.md](deployment.md) — audit-DB PVC mount at `/data` for in-cluster deployments
+- [development.md](development.md) — audit configuration section in the developer guide

From 52215786c502925d4dada0ff61c1dbe121f8c247 Mon Sep 17 00:00:00 2001
From: Melvin Hillsman <mhillsma@redhat.com>
Date: Fri, 22 May 2026 10:37:25 -0500
Subject: [PATCH 16/17] docs: add configuration.md as the complete config
 reference
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Configuration was documented in three places (README env-var table,
development.md SSH/audit sections, deploy/configmap.yaml comments) and
none of them was complete. Per-workload params (tps file-size /
iterations / duration; chaos-disk fill-percent etc.) were not documented
at all, and the per-workload YAML override surface was opaque.

Consolidate everything into a single reference:

- Three full flag tables (global persistent, run, cleanup) with flag /
  env var / YAML key / default / description per row
- One alphabetical environment-variable table, including the two
  deployment-only entrypoint vars (VIRTWORK_COMMAND, VIRTWORK_ARGS) with
  a callout that those are read by entrypoint.sh, not by Viper
- A full YAML config schema covering every key including per-workload
  overrides for all nine workloads, with comments showing default values
- A per-workload params table (tps file-size/iterations/duration;
  chaos-disk mount/fill-percent/fill-sleep/release-sleep;
  chaos-network latency-ms/packet-loss-percent;
  chaos-process signal/interval/min-pid) — documented as the spec for
  user-tunable parameters, including the ones not yet wired through
  WorkloadConfig.Params
- A ConfigMap-key reference matching the in-cluster defaults from
  deploy/configmap.yaml
- A priority-chain mermaid + a step-by-step walkthrough of how
  internal/config/config.go LoadConfig applies the chain (including the
  --no-audit short-circuit handled outside the chain)
- A contributor section explaining the mapstructure tag convention and
  where to wire new params

Cross-link from chaos-workloads, deployment, development, audit-schema.

Signed-off-by: Melvin Hillsman <mhillsma@redhat.com>
---
 docs/configuration.md | 268 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 268 insertions(+)
 create mode 100644 docs/configuration.md

diff --git a/docs/configuration.md b/docs/configuration.md
new file mode 100644
index 0000000..19e3d1c
--- /dev/null
+++ b/docs/configuration.md
@@ -0,0 +1,268 @@
+# Configuration Reference
+
+Virtwork configuration flows through four sources, merged in priority order:
+
+```
+CLI flags  >  environment variables (VIRTWORK_*)  >  YAML config file  >  built-in defaults
+```
+
+This document is the authoritative reference: every flag, every environment variable, every YAML key, every ConfigMap entry, and every per-workload parameter. For the *summary*, see the top-level [README](../README.md#configuration). For the implementation, see `internal/config/config.go`.
+
+## Quick Reference
+
+| Surface | When to use |
+|---|---|
+| **CLI flags** | One-off overrides; whatever you type wins |
+| **Environment variables** (`VIRTWORK_*`) | CI/CD pipelines; ConfigMap-driven in-cluster deployments |
+| **YAML config file** (`--config path.yaml`) | Reproducible runs; expressing per-workload overrides |
+| **Defaults** | Sensible behavior with no arguments at all |
+
+---
+
+## Global Persistent Flags
+
+These apply to both `virtwork run` and `virtwork cleanup`.
+
+| Flag | Env Var | YAML Key | Default | Description |
+|---|---|---|---|---|
+| `--namespace` | `VIRTWORK_NAMESPACE` | `namespace` | `virtwork` | Kubernetes namespace for VMs and supporting resources |
+| `--kubeconfig` | `VIRTWORK_KUBECONFIG` | `kubeconfig` | _empty (in-cluster, then `~/.kube/config`)_ | Path to kubeconfig file |
+| `--config` | — | — | _empty_ | Path to a YAML config file to merge in |
+| `--verbose` | `VIRTWORK_VERBOSE` | `verbose` | `false` | Switch the logger from INFO to DEBUG |
+| `--audit` | `VIRTWORK_AUDIT` | `audit` | `true` | Enable SQLite audit tracking |
+| `--no-audit` | — | — | _unset_ | Hard override: disable audit tracking regardless of `--audit` / `VIRTWORK_AUDIT` |
+| `--audit-db` | `VIRTWORK_AUDIT_DB` | `audit_db` | `virtwork.db` | Path to the audit SQLite file (use `/data/virtwork.db` in-cluster) |
+
+**Audit precedence note:** `--no-audit` short-circuits everything. Otherwise, the `audit` boolean follows the standard chain. In-cluster, set `VIRTWORK_AUDIT=false` in the ConfigMap to disable.
+
+---
+
+## `virtwork run` Flags
+
+| Flag | Env Var | YAML Key | Default | Description |
+|---|---|---|---|---|
+| `--workloads` | — | _(handled via `workloads` map per-key)_ | All nine workloads, sorted: `chaos-disk, chaos-network, chaos-process, cpu, database, disk, memory, network, tps` | Comma-separated list of workloads to deploy |
+| `--vm-count` | — | `workloads.<name>.vm_count` | `1` | VMs per workload (multi-VM workloads multiply by `len(Roles())`, so 1 becomes 2 for network and tps) |
+| `--cpu-cores` | `VIRTWORK_CPU_CORES` | `cpu_cores` / `workloads.<name>.cpu_cores` | `2` | CPU cores per VM (per-workload override beats global) |
+| `--memory` | `VIRTWORK_MEMORY` | `memory` / `workloads.<name>.memory` | `2Gi` | Memory per VM |
+| `--disk-size` | `VIRTWORK_DATA_DISK_SIZE` | `data_disk_size` | `10Gi` | Data-disk size for storage-backed workloads (disk, database, chaos-disk) |
+| `--container-disk-image` | `VIRTWORK_CONTAINER_DISK_IMAGE` | `container_disk_image` | `quay.io/containerdisks/fedora:41` | Boot image for the VMs; set to the golden image for faster boot |
+| `--dry-run` | `VIRTWORK_DRY_RUN` | `dry_run` | `false` | Print the VM YAML and exit; do not connect to a cluster |
+| `--no-wait` | — | (sets `wait_for_ready=false`) | _unset_ | Skip readiness polling after VMs are created |
+| `--timeout` | `VIRTWORK_TIMEOUT` | `timeout` | `600` | Readiness timeout in seconds |
+| `--ssh-user` | `VIRTWORK_SSH_USER` | `ssh_user` | `virtwork` | Username for the in-VM user account (only created when at least one SSH credential is provided) |
+| `--ssh-password` | `VIRTWORK_SSH_PASSWORD` | `ssh_password` | _empty_ | Password for the SSH user (plain-text in the VM spec — prefer keys) |
+| `--ssh-key` | — | `ssh_authorized_keys` (YAML list) | _empty_ | Inline SSH public key; repeatable |
+| `--ssh-key-file` | — | _(handled as inline key after reading)_ | _empty_ | Path to a public key file; repeatable |
+| _(env only)_ | `VIRTWORK_SSH_AUTHORIZED_KEYS` | `ssh_authorized_keys` | _empty_ | Comma-separated list of inline keys (env-var form) |
+
+---
+
+## `virtwork cleanup` Flags
+
+| Flag | Env Var | YAML Key | Default | Description |
+|---|---|---|---|---|
+| `--delete-namespace` | — | — | `false` | Also delete the namespace itself after deleting managed resources |
+| `--run-id` | — | — | _empty_ | Limit cleanup to resources labeled `virtwork/run-id=<uuid>` |
+| `--dry-run` | — | — | `false` | Print intent without actually deleting |
+
+Global flags (`--namespace`, `--kubeconfig`, `--audit`, `--no-audit`, `--audit-db`, `--verbose`) also apply.
+
+---
+
+## Environment Variables (Alphabetical)
+
+| Variable | Type | Default | Source | Description |
+|---|---|---|---|---|
+| `VIRTWORK_AUDIT` | bool | `true` | flag-bound | Enable audit tracking |
+| `VIRTWORK_AUDIT_DB` | string | `virtwork.db` | flag-bound | Audit SQLite path |
+| `VIRTWORK_CONTAINER_DISK_IMAGE` | string | `quay.io/containerdisks/fedora:41` | flag-bound | VM boot image |
+| `VIRTWORK_CPU_CORES` | int | `2` | flag-bound | Default per-VM CPU cores |
+| `VIRTWORK_DATA_DISK_SIZE` | string | `10Gi` | flag-bound | Default data disk size |
+| `VIRTWORK_DRY_RUN` | bool | `false` | flag-bound | Default for `--dry-run` |
+| `VIRTWORK_KUBECONFIG` | string | _empty_ | flag-bound | Kubeconfig path |
+| `VIRTWORK_MEMORY` | string | `2Gi` | flag-bound | Default per-VM memory |
+| `VIRTWORK_NAMESPACE` | string | `virtwork` | flag-bound | Target namespace |
+| `VIRTWORK_SSH_AUTHORIZED_KEYS` | string (csv) | _empty_ | env-only | Comma-separated SSH public keys |
+| `VIRTWORK_SSH_PASSWORD` | string | _empty_ | flag-bound | SSH password |
+| `VIRTWORK_SSH_USER` | string | `virtwork` | flag-bound | SSH username |
+| `VIRTWORK_TIMEOUT` | int | `600` | flag-bound | Readiness timeout seconds |
+| `VIRTWORK_VERBOSE` | bool | `false` | flag-bound | Verbose logging |
+| `VIRTWORK_WAIT_FOR_READY` | bool | `true` | flag-bound | Inverse of `--no-wait` |
+| `VIRTWORK_COMMAND` | string | _empty_ | deployment only | In-pod auto-run command: `run`, `cleanup`, or empty (sleep). Read by `entrypoint.sh`, not Viper. |
+| `VIRTWORK_ARGS` | string | _empty_ | deployment only | Extra arguments when `VIRTWORK_COMMAND` is set. Read by `entrypoint.sh`, not Viper. |
+
+`VIRTWORK_COMMAND` and `VIRTWORK_ARGS` are container entrypoint behavior — they tell `entrypoint.sh` whether to run virtwork immediately or to sleep until invoked via `oc exec`. See [deployment.md](deployment.md).
+
+---
+
+## YAML Config File
+
+Full schema with every supported key:
+
+```yaml
+# Global defaults
+namespace: virtwork-prod
+container_disk_image: quay.io/opdev/virtwork-disk:latest    # set to golden image to skip first-boot package installs
+data_disk_size: 20Gi
+cpu_cores: 2
+memory: 2Gi
+
+# Cluster connection (omit to use in-cluster or ~/.kube/config)
+kubeconfig: /etc/virtwork/kubeconfig
+
+# Behavior
+dry_run: false
+verbose: false
+wait_for_ready: true
+timeout: 900
+
+# SSH (optional — when omitted, no user account is created in the VM)
+ssh_user: virtwork
+ssh_password: ""              # prefer keys
+ssh_authorized_keys:
+  - ssh-ed25519 AAAAC3Nz... key-1
+  - ssh-ed25519 AAAAC3Nz... key-2
+
+# Audit
+audit: true
+audit_db: /data/virtwork.db
+
+# Per-workload overrides (everything optional; unspecified keys inherit globals)
+workloads:
+  cpu:
+    enabled: true
+    vm_count: 2
+    cpu_cores: 4
+    memory: 4Gi
+  memory:
+    enabled: true
+    vm_count: 1
+  disk:
+    enabled: true
+  database:
+    enabled: true
+    cpu_cores: 2
+    memory: 4Gi
+  network:
+    enabled: true
+    vm_count: 1                # creates 1 server + 1 client = 2 VMs
+  tps:
+    enabled: true
+    vm_count: 1                # creates 1 server + 1 client = 2 VMs
+    params:
+      file-size: "50M"         # default 10M
+      iterations: "10"         # default 30
+      duration: "30"           # default 60 (seconds per iteration)
+  chaos-disk:
+    enabled: true
+    params:
+      mount: /mnt/data         # default /mnt/data
+      fill-percent: "80"       # default 90
+      fill-sleep: "120"        # default 60
+      release-sleep: "60"      # default 30
+  chaos-network:
+    enabled: true
+    params:
+      latency-ms: "250"        # default 100
+      packet-loss-percent: "10"  # default 5.0
+  chaos-process:
+    enabled: true
+    params:
+      signal: "SIGKILL"        # default SIGTERM
+      interval: "15"           # default 30
+      min-pid: "500"           # default 1000
+```
+
+### Per-workload `params` keys
+
+Each workload's `params` block accepts string-valued keys. The current parameters per workload:
+
+| Workload | Key | Default | Effect |
+|---|---|---|---|
+| **tps** | `file-size` | `10M` | Size of the HTTP test file (suffix `K`/`M`/`G`) |
+| **tps** | `iterations` | `30` | Number of test iterations |
+| **tps** | `duration` | `60` | Seconds per iteration |
+| **chaos-disk** | `mount` | `/mnt/data` | Mountpoint of the data disk to fill |
+| **chaos-disk** | `fill-percent` | `90` | Target fill percentage |
+| **chaos-disk** | `fill-sleep` | `60` | Seconds held at target fill |
+| **chaos-disk** | `release-sleep` | `30` | Seconds empty before refilling |
+| **chaos-network** | `latency-ms` | `100` | `netem` egress delay |
+| **chaos-network** | `packet-loss-percent` | `5.0` | `netem` egress drop rate |
+| **chaos-process** | `signal` | `SIGTERM` | Signal sent to victims |
+| **chaos-process** | `interval` | `30` | Seconds between kills |
+| **chaos-process** | `min-pid` | `1000` | Minimum PID considered eligible |
+
+Workloads without entries above accept no per-workload `params` today (cpu, memory, disk, database, network).
+
+---
+
+## ConfigMap for In-Cluster Deployment
+
+`deploy/configmap.yaml` ships the following defaults. Edit and `oc apply -k deploy/` to change behavior of an in-cluster pod.
+
+| ConfigMap Key | Default | Notes |
+|---|---|---|
+| `VIRTWORK_NAMESPACE` | `virtwork` | The pod creates VMs in this namespace |
+| `VIRTWORK_CONTAINER_DISK_IMAGE` | `quay.io/containerdisks/fedora:41` | Change to the golden image to speed up boot |
+| `VIRTWORK_DATA_DISK_SIZE` | `10Gi` | |
+| `VIRTWORK_CPU_CORES` | `2` | |
+| `VIRTWORK_MEMORY` | `2Gi` | |
+| `VIRTWORK_WAIT_FOR_READY` | `true` | |
+| `VIRTWORK_TIMEOUT` | `600` | |
+| `VIRTWORK_DRY_RUN` | `false` | |
+| `VIRTWORK_VERBOSE` | `false` | |
+| `VIRTWORK_AUDIT` | `true` | |
+| `VIRTWORK_AUDIT_DB` | `/data/virtwork.db` | Mounted from the `virtwork-audit-data` PVC |
+| `VIRTWORK_SSH_USER` | `virtwork` | |
+
+`VIRTWORK_COMMAND` and `VIRTWORK_ARGS` are set directly in the Deployment `env:` (not the ConfigMap) so that updating them triggers a pod restart. See [deployment.md](deployment.md).
+
+`deploy/secret.yaml` provides `VIRTWORK_SSH_PASSWORD` and `VIRTWORK_SSH_AUTHORIZED_KEYS` separately so that credentials are not in the ConfigMap.
+
+---
+
+## Priority Chain in Detail
+
+```mermaid
+flowchart TD
+    CLI["CLI flag explicitly set\ne.g. --namespace virtwork-test"]
+    ENV["Environment variable\ne.g. VIRTWORK_NAMESPACE=virtwork-prod"]
+    YAML["YAML config file\nnamespace: virtwork-staging"]
+    DEF["Built-in default\n'virtwork'"]
+
+    CLI -->|wins| MERGE[Effective Config]
+    ENV -->|2nd| MERGE
+    YAML -->|3rd| MERGE
+    DEF -->|4th| MERGE
+```
+
+How it works in code (`internal/config/config.go` → `LoadConfig`):
+
+1. `SetDefaults(v)` seeds Viper with the built-in defaults.
+2. `v.SetEnvPrefix("VIRTWORK")` + `v.AutomaticEnv()` enables automatic env-var binding (with `-` ↔ `_` replacement so `--data-disk-size` ↔ `VIRTWORK_DATA_DISK_SIZE`).
+3. If `--config <path>` is set, `v.ReadInConfig()` merges the YAML file (overrides env and defaults).
+4. `bindFlagIfSet(...)` walks each known flag and copies the value into Viper *only if* the flag was explicitly set on the command line. This makes CLI flags the top of the stack without clobbering env/YAML values from defaults.
+5. The `Config` struct is populated from Viper's effective view.
+
+Workload-specific config (`workloads.<name>.*`) is parsed as a separate map in step 5 via `v.UnmarshalKey("workloads", ...)` — it only exists in YAML; there is no env-var or flag form.
+
+The `--no-audit` flag is checked separately by `initAuditor` in `cmd/virtwork/main.go` and short-circuits the standard chain.
+
+---
+
+## For Contributors
+
+- All config field definitions live in `internal/config/config.go` — `WorkloadConfig` struct, `Config` struct, `SetDefaults`, `BindFlags`, `LoadConfig`.
+- The mapstructure tag on each field is the YAML key name (e.g., `mapstructure:"data-disk-size"` → YAML key `data_disk_size` after the `_`/`-` replacement). Match this convention when adding new fields.
+- Per-workload `params` are surfaced into the workload as `WorkloadConfig.Params map[string]string`. New chaos / multi-VM knobs go here so they're uniformly addressable from YAML.
+- When adding a new env var, prefer letting Viper bind it automatically rather than hand-rolling `os.Getenv` (see `VIRTWORK_SSH_AUTHORIZED_KEYS` in `resolveSSHKeys` for the one current exception, driven by the comma-split list semantics).
+- Update this document, the ConfigMap default list, and the per-workload `params` table whenever you add a new knob.
+
+## Related Docs
+
+- [README.md](../README.md#configuration) — configuration summary in the main project README
+- [chaos-workloads.md](chaos-workloads.md) — narrative description of how chaos params shape workload behavior
+- [deployment.md](deployment.md) — ConfigMap + Secret + Deployment env semantics
+- [development.md](development.md) — adding new workloads (including new params)
+- [audit-schema.md](audit-schema.md) — audit-related configuration consequences

From 6039be49f7b1108de5505261d28d48407d8c497c Mon Sep 17 00:00:00 2001
From: Melvin Hillsman <mhillsma@redhat.com>
Date: Fri, 22 May 2026 10:38:53 -0500
Subject: [PATCH 17/17] docs: add deployment.md as the Kustomize deployment
 deep-dive

The README's OpenShift Deployment section is a quick orientation; the
full surface (ConfigMap keys, RBAC scope, entrypoint behavior, PVC
ownership, image pinning policy, security context, DataVolume name
observation) was not documented.

Cover the full deployment story:

- A mermaid topology diagram showing Namespace + ServiceAccount + RBAC
  + ConfigMap + Secret + PVC + Deployment + Pod relationships
- A manifest-by-manifest table with file path and purpose
- VIRTWORK_COMMAND and VIRTWORK_ARGS semantics: how entrypoint.sh
  dispatches between run / cleanup / sleep, including the unquoted-args
  constraint (no spaces inside argument values)
- Interactive vs auto-execute deployment patterns
- Image pinning policy (semver vs :latest, imagePullPolicy implications,
  podman build commands, multi-stage Dockerfile + Dockerfile.ci)
- Using the golden image by setting VIRTWORK_CONTAINER_DISK_IMAGE in
  the ConfigMap
- Full RBAC scope table explaining why each verb is needed (and why
  the role is cluster-scoped because Namespaces are cluster-scoped)
- Sizing guidance and the note that VM resource sizing happens at the
  VirtualMachine spec, not the virtwork pod limits
- Why DataVolume names look like virtwork-disk-data-virtwork-disk-0
  (the namespaceDataVolumes suffix from cmd/virtwork/main.go that
  prevents collisions across multiple VMs of the same workload)
- Audit DB persistence on a separate PVC so upgrades preserve history,
  with copy-out and in-place query patterns
- Security context defaults (runAsNonRoot, RuntimeDefault seccomp,
  capabilities ALL dropped) so reviewers know what's in place
- Uninstall commands distinguishing pod-only vs full removal

Cross-link from README, configuration.md, audit-schema.md,
chaos-workloads.md, and the golden-image README.

Signed-off-by: Melvin Hillsman <mhillsma@redhat.com>
---
 docs/deployment.md | 226 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 226 insertions(+)
 create mode 100644 docs/deployment.md

diff --git a/docs/deployment.md b/docs/deployment.md
new file mode 100644
index 0000000..07670b6
--- /dev/null
+++ b/docs/deployment.md
@@ -0,0 +1,226 @@
+# OpenShift Deployment
+
+Virtwork can run as a pod on the cluster it operates against, using the Kustomize manifests in [`deploy/`](../deploy). This is the right deployment model when you want virtwork to be reachable from `oc exec`, scheduled by CI, or driven by a GitOps workflow.
+
+For the smaller "run from my laptop with my kubeconfig" mode, just build and execute the binary — no manifests are needed.
+
+## What `oc apply -k deploy/` Creates
+
+```mermaid
+flowchart TB
+    NS["Namespace<br/>virtwork"]
+
+    subgraph Identity
+        SA["ServiceAccount<br/>virtwork"]
+        CR["ClusterRole<br/>virtwork<br/>(VMs, VMIs, DVs,<br/>Services, Secrets, Namespaces)"]
+        CRB["ClusterRoleBinding<br/>virtwork"]
+    end
+
+    subgraph Config
+        CM["ConfigMap<br/>virtwork-config<br/>VIRTWORK_* defaults"]
+        SEC["Secret<br/>virtwork-secrets<br/>VIRTWORK_SSH_PASSWORD<br/>VIRTWORK_SSH_AUTHORIZED_KEYS"]
+    end
+
+    subgraph Persistence
+        PVC["PersistentVolumeClaim<br/>virtwork-audit-data<br/>RWO, 1Gi"]
+    end
+
+    subgraph Workload
+        DEP["Deployment<br/>virtwork<br/>replicas: 1, Recreate strategy"]
+        POD["Pod<br/>(entrypoint.sh)"]
+    end
+
+    SA --> CRB
+    CR --> CRB
+    CRB --> SA
+
+    NS --> SA
+    NS --> CM
+    NS --> SEC
+    NS --> PVC
+    NS --> DEP
+    DEP --> POD
+    SA --> POD
+    CM --> POD
+    SEC --> POD
+    PVC --> POD
+```
+
+| Resource | File | Purpose |
+|---|---|---|
+| Namespace `virtwork` | `namespace.yaml` | Isolation boundary for virtwork-managed resources |
+| ServiceAccount `virtwork` | `serviceaccount.yaml` | Identity the pod runs as |
+| ClusterRole + ClusterRoleBinding `virtwork` | `rbac.yaml` | Permissions to manage VMs, VMIs, DataVolumes, Services, Secrets, Namespaces — cluster-scoped because Namespaces themselves are cluster-scoped |
+| ConfigMap `virtwork-config` | `configmap.yaml` | Default `VIRTWORK_*` env vars |
+| Secret `virtwork-secrets` | `secret.yaml` | SSH credentials (kept out of the ConfigMap) |
+| PersistentVolumeClaim `virtwork-audit-data` | `pvc.yaml` | 1Gi RWO claim mounted at `/data` for the audit SQLite file |
+| Deployment `virtwork` | `deployment.yaml` | The pod itself |
+
+The Kustomization (`deploy/kustomization.yaml`) applies `app.kubernetes.io/name: virtwork` and `app.kubernetes.io/managed-by: virtwork` labels to every resource.
+
+## Deploy
+
+```bash
+oc apply -k deploy/
+
+# Verify
+oc get all,configmap,secret,pvc -n virtwork
+```
+
+The Deployment uses `strategy.type: Recreate` because the audit PVC is `ReadWriteOnce` — a rolling update would attempt to schedule two pods simultaneously and the new pod would fail to bind the volume.
+
+## Container Behavior: `VIRTWORK_COMMAND` and `VIRTWORK_ARGS`
+
+The container's behavior is governed by two environment variables defined directly in the Deployment (not the ConfigMap, so that changing them rolls the pod):
+
+| Env | Effect |
+|---|---|
+| `VIRTWORK_COMMAND` empty (default) | Pod runs `sleep infinity`. Use `oc exec` to run virtwork commands by hand. |
+| `VIRTWORK_COMMAND=run` | Pod auto-executes `virtwork run $VIRTWORK_ARGS` on start. |
+| `VIRTWORK_COMMAND=cleanup` | Pod auto-executes `virtwork cleanup $VIRTWORK_ARGS` on start. |
+
+The entrypoint script ([`entrypoint.sh`](../entrypoint.sh)) validates `VIRTWORK_COMMAND` (only `run` and `cleanup` are accepted) and `exec`s into virtwork. Pod stdout/stderr ends up in the container log.
+
+**Important:** `VIRTWORK_ARGS` is intentionally unquoted to allow space-delimited tokens (`"--workloads cpu,memory --vm-count 2"`). As a consequence, **no argument value can contain a space.**
+
+### Interactive use
+
+```bash
+oc exec -it deploy/virtwork -- virtwork run --dry-run
+oc exec -it deploy/virtwork -- virtwork run --workloads cpu,memory
+oc exec -it deploy/virtwork -- virtwork cleanup
+oc exec -it deploy/virtwork -- sqlite3 /data/virtwork.db "SELECT run_id, command, status, started_at FROM audit_log ORDER BY id DESC LIMIT 5"
+```
+
+### Auto-execute on pod start
+
+```bash
+oc set env deploy/virtwork VIRTWORK_COMMAND=run VIRTWORK_ARGS="--workloads cpu,memory --vm-count 2"
+```
+
+The deployment rolls the pod, which executes virtwork and exits. The audit row persists on the PVC. For repeatable scheduled runs, drive `VIRTWORK_COMMAND` / `VIRTWORK_ARGS` from a CronJob or GitOps tool rather than the Deployment directly.
+
+## Image Pinning
+
+The Deployment ships pinned to a semantic version (`quay.io/opdev/virtwork:v0.0.1`) for reproducibility. Bump the tag when upgrading.
+
+```yaml
+containers:
+  - name: virtwork
+    image: quay.io/opdev/virtwork:v0.0.1   # update on upgrade
+    imagePullPolicy: IfNotPresent
+```
+
+For development you may pin to `:latest` instead, but be aware that `IfNotPresent` plus `:latest` can pin a node to an older copy of the image until it's pulled again. Either bump the tag, or set `imagePullPolicy: Always` for floating tags.
+
+The container image is built from [`Dockerfile`](../Dockerfile) (multi-stage: `golang:1.25-bookworm` builder for CGO + `ubi9/ubi-minimal` runtime with `sqlite-libs`). A CI variant lives at [`Dockerfile.ci`](../Dockerfile.ci).
+
+```bash
+podman build -t quay.io/opdev/virtwork:vX.Y.Z .
+podman push quay.io/opdev/virtwork:vX.Y.Z
+```
+
+Then update `deploy/deployment.yaml` and `oc apply -k deploy/`.
+
+### Using the Golden Container Disk Image
+
+The golden image at `quay.io/opdev/virtwork-disk:latest` pre-installs every workload's tools so VMs boot ready-to-run. To make it the default for an in-cluster deployment, set the ConfigMap key:
+
+```bash
+oc set data configmap/virtwork-config -n virtwork VIRTWORK_CONTAINER_DISK_IMAGE=quay.io/opdev/virtwork-disk:latest
+```
+
+The Deployment uses `envFrom` to pull all ConfigMap keys into the pod, so the change takes effect on the next `virtwork run`. No pod restart needed. See [../build/golden-image/README.md](../build/golden-image/README.md) for build instructions.
+
+## RBAC Scope
+
+The ClusterRole `virtwork` grants:
+
+| API Group | Resources | Verbs | Why |
+|---|---|---|---|
+| `""` (core) | `namespaces` | `create`, `get`, `delete` | `EnsureNamespace`; `cleanup --delete-namespace` |
+| `kubevirt.io` | `virtualmachines` | `create`, `delete`, `get`, `list` | Full VM lifecycle |
+| `kubevirt.io` | `virtualmachineinstances` | `get`, `list` | Readiness polling (`GetVMIPhase`) |
+| `cdi.kubevirt.io` | `datavolumes` | `create`, `delete`, `get`, `list` | DataVolumes are created via embedded `DataVolumeTemplates` in VMs |
+| `""` (core) | `services` | `create`, `delete`, `get`, `list` | Multi-VM workloads (network, tps) need ClusterIP Services |
+| `""` (core) | `secrets` | `create`, `delete`, `get`, `list` | Cloud-init userdata Secrets (one per VM) |
+
+This is a ClusterRole — not a Role — because `namespaces` are cluster-scoped. Permissions are intentionally broad to support cross-namespace test setups; tighten the binding to a single namespace if your environment requires it.
+
+## Sizing
+
+The Deployment's defaults (`requests: 64Mi / 100m`, `limits: 256Mi / 500m`) are sized for the orchestrator process itself, not the VMs it creates. Bump the limits if:
+
+- You deploy all nine workloads simultaneously (11 VMs of plumbing in one pod) — concurrent VM-create goroutines and audit writes consume modestly more memory.
+- You consume the audit DB with long-running queries from within the pod.
+
+The VMs themselves are sized via `--cpu-cores`, `--memory`, and per-workload YAML overrides; their resource requests come from the VirtualMachine spec, not the virtwork pod.
+
+## DataVolume Names
+
+When you list DataVolumes inside the namespace, you'll see entries like:
+
+```
+$ oc get dv -n virtwork
+NAME                             PHASE       AGE
+virtwork-disk-data-virtwork-disk-0          Succeeded   3m
+virtwork-database-data-virtwork-database-0  Succeeded   3m
+virtwork-chaos-disk-data-virtwork-chaos-disk-0  Succeeded  3m
+```
+
+The suffix (`-virtwork-disk-0`, `-virtwork-chaos-disk-0`) comes from `namespaceDataVolumes` in `cmd/virtwork/main.go`. Each `VMCount > 1` deployment gets its own uniquely-named DataVolume so multiple VMs of the same workload type don't collide on a single namespace-scoped DV name. The base name (`virtwork-disk-data`, etc.) is what the workload declares in its `DataVolumeTemplates()` method; the orchestrator appends the VM name.
+
+## Persisting the Audit Database
+
+The audit DB lives at `/data/virtwork.db` in-pod, mounted from the `virtwork-audit-data` PVC. The default PVC is 1Gi RWO — plenty for tens of thousands of executions.
+
+The PVC is intentionally separate from the Deployment so that `oc apply -k deploy/` upgrades preserve audit history.
+
+To inspect from outside the cluster:
+
+```bash
+# Copy the DB out
+oc cp virtwork/$(oc get pod -n virtwork -l app.kubernetes.io/name=virtwork -o name | head -1 | cut -d/ -f2):/data/virtwork.db ./virtwork.db
+sqlite3 virtwork.db "SELECT run_id, command, status, started_at FROM audit_log ORDER BY id DESC LIMIT 10"
+```
+
+Or query in-place:
+
+```bash
+oc exec -n virtwork deploy/virtwork -- sqlite3 /data/virtwork.db "SELECT run_id, command, status, started_at FROM audit_log ORDER BY id DESC LIMIT 10"
+```
+
+See [audit-schema.md](audit-schema.md) for the full schema and more queries.
+
+## Security Context
+
+The pod runs:
+
+- `runAsNonRoot: true`
+- `seccompProfile.type: RuntimeDefault`
+- `allowPrivilegeEscalation: false`
+- `capabilities.drop: [ALL]`
+
+These are pod- and container-level defaults that the security review process expects. Override only if you have a specific need.
+
+The Secret (`virtwork-secrets`) carries the SSH password and authorized-keys list. It is mounted via `envFrom` so the values appear as env vars inside the pod. **Do not store SSH passwords as plain text in production** — prefer authorized keys via `VIRTWORK_SSH_AUTHORIZED_KEYS`, and even then treat the secret as sensitive in your cluster RBAC.
+
+## Uninstalling
+
+```bash
+# Just the pod and identity, keeping the audit history
+oc delete deployment virtwork -n virtwork
+
+# Or the whole thing including the namespace and PVC
+oc delete -k deploy/
+```
+
+`oc delete -k deploy/` removes everything in the Kustomize set, including the PVC. If you want to keep the audit history across uninstalls, omit `pvc.yaml` from the deletion or delete resources individually.
+
+## Related Docs
+
+- [README.md](../README.md#openshift-deployment) — the high-level deployment summary
+- [configuration.md](configuration.md) — every ConfigMap key with its meaning and default
+- [audit-schema.md](audit-schema.md) — what's in the PVC-backed SQLite file
+- [chaos-workloads.md](chaos-workloads.md) — running chaos workloads from an in-cluster pod
+- [../build/golden-image/README.md](../build/golden-image/README.md) — building and using the optional golden container disk image