feat: golden container disk image with pre-installed workload and chaos engineering tools

[mrhillsman]

Severity

Low

Description

Virtwork currently installs workload tools (stress-ng, fio, iperf3, pgbench) at VM boot time via cloud-init package installation. This approach adds boot latency, depends on external package repositories being available, and increases the chance of transient failures during provisioning.

This issue proposes building a pre-baked golden container disk image with all workload and chaos engineering tools pre-installed. VMs would boot ready to run workloads immediately, with no package installation step required.

In addition to the existing workload generators, the golden image should include chaos engineering tools to enable fault injection and resilience testing scenarios:

• tc (traffic control) — Network chaos: latency injection, packet loss, bandwidth throttling
• iptables — Network partition simulation, port blocking
• stress-ng — Already used for CPU/memory workloads; also supports thermal, I/O, and scheduler stress
• fio — Already used for disk workloads; can simulate disk saturation scenarios
• iperf3 — Already used for network workloads
• pgbench — Already used for database workloads
• kill / pkill — Process chaos (available by default in base image)
• fallocate / dd — Disk fill chaos
• tc-netem — Advanced network emulation (jitter, reordering, duplication)

Files to Modify or Create

New files

• build/golden-image/Containerfile — Multi-stage build definition for the golden container disk image based on Fedora 41 with all tools pre-installed
• internal/workloads/chaos_network.go — Network chaos workload (tc/netem-based latency, packet loss, bandwidth limiting)
• internal/workloads/chaos_process.go — Process chaos workload (random process killing, OOM triggering)
• internal/workloads/chaos_disk.go — Disk fill chaos workload (fallocate/dd-based disk pressure)

Modified files

• internal/workloads/cpu.go — Remove dnf install from cloud-init script; assume stress-ng is pre-installed
• internal/workloads/memory.go — Remove dnf install from cloud-init script; assume stress-ng is pre-installed
• internal/workloads/database.go — Remove dnf install from cloud-init script; assume postgresql-server and pgbench are pre-installed
• internal/workloads/network.go — Remove dnf install from cloud-init script; assume iperf3 is pre-installed
• internal/workloads/disk.go — Remove dnf install from cloud-init script; assume fio is pre-installed
• internal/workloads/registry.go — Register new chaos workloads (chaos-network, chaos-process, chaos-disk)
• internal/config/config.go — Add configuration options for chaos workload parameters (e.g., latency amount, packet loss percentage, disk fill target)
• internal/constants/defaults.go — Add default values for chaos workload parameters and update default container disk image to the golden image

Expected

Golden Image Build

• A Containerfile in build/golden-image/ that extends quay.io/containerdisks/fedora:41 and installs all required packages: stress-ng, fio, iperf3, postgresql-server, iproute-tc, iptables-nft, and their dependencies
• The image is published to a container registry (e.g., quay.io/opdev/virtwork-golden:latest)
• The default --container-disk-image value is updated to point to the golden image

Workload Changes

• All existing workload cloud-init scripts are simplified to remove package installation steps (dnf install blocks)
• Cloud-init scripts only configure and start systemd services, since all binaries are already present
• Boot time is reduced significantly as no packages need to be downloaded or installed

Chaos Engineering Workloads

• New workloads follow the existing Workload interface pattern defined in internal/workloads/workload.go
• Each chaos workload is registered in the workload registry and selectable via --workloads chaos-network,chaos-disk,chaos-process
• chaos-network: Uses tc qdisc and netem to inject configurable latency (default 100ms), packet loss (default 5%), and bandwidth limits; runs as a systemd service with configurable duration and interval
• chaos-process: Periodically sends signals to random non-essential processes; configurable signal type and interval
• chaos-disk: Uses fallocate to fill a target mount point to a configurable percentage (default 90%); runs in a loop with fill/release cycles
• All chaos workloads support --vm-count for running across multiple VMs simultaneously

[mrhillsman]

This issue has been split into focused, template-aligned issues:

• [Feature]: Golden container disk image with pre-installed workload tools #16 — [Feature]: Golden container disk image with pre-installed workload tools
• [Workload]: chaos-network workload (tc/netem-based latency, packet loss, bandwidth limiting) #17 — [Workload]: chaos-network workload (tc/netem-based)
• [Workload]: chaos-process workload (random process killing) #18 — [Workload]: chaos-process workload (random process killing)
• [Workload]: chaos-disk workload (disk fill pressure) #19 — [Workload]: chaos-disk workload (disk fill pressure)

All original content has been preserved across the replacement issues.