[Bug]: PVC-backed workloads fail to discover, format, and mount data disks reliably

[mrhillsman]

Bug Description

Three workloads that attach an extra PVC-backed data disk via KubeVirt DataVolumeTemplates — chaos-disk, disk (fio), and database — have the same fundamental problem: the extra disk arrives as a raw, unformatted virtio block device, but the cloud-init scripts either never partition/format/mount it (chaos-disk, disk) or hardcode the device path as /dev/vdc (database).

The hardcoded path is fragile because virtio disks are named vdX based on attachment order (rootdisk=vda, cloudinit=vdb, data=vdc), but this can shift if other virtio disks are added or reordered.

Steps to Reproduce

1. Run virtwork run --workloads chaos-disk or virtwork run --workloads disk or virtwork run --workloads database
2. Wait for VMs to become ready
3. SSH into the VM and check /mnt/data (chaos-disk/disk) or /var/lib/pgsql/data (database)
4. Observe: the data disk is not formatted or mounted (chaos-disk/disk), or the format/mount fails if disk ordering changes (database)

Expected Behavior

The data disk should be reliably discovered, formatted with xfs, and mounted at the expected path (/mnt/data for chaos-disk/disk, /var/lib/pgsql/data for database) regardless of virtio device letter ordering.

Actual Behavior

• chaos-disk: mkdir -p /mnt/data runs but the PVC-backed disk is never formatted or mounted — the workload writes to the rootfs instead
• disk (fio): Same — mkdir -p /mnt/data creates a directory on the rootfs, fio writes to rootfs not the data disk
• database: Hardcodes /dev/vdc which is fragile — if disk ordering changes the format/mount fails and PostgreSQL cannot start

Command

virtwork run

Workload Type

• disk
• database
• chaos-disk (not in dropdown but affected)

Additional Context

Root cause: No Serial field is set on the kubevirtv1.Disk structs in ExtraDisks(). KubeVirt exposes serials inside the guest at /dev/disk/by-id/virtio-<serial>, providing a stable symlink regardless of device letter. Without a serial, there is no reliable way to discover the data disk inside the guest.

Fix: Set a deterministic Serial on each workload's ExtraDisks() disk, then update cloud-init to discover the disk by serial, format with xfs, and mount before the workload service starts. See docs/plans/fix-disks-for-workloads.md for the full implementation plan.

Affected files:

• internal/workloads/chaos_disk.go
• internal/workloads/disk.go
• internal/workloads/database.go
• internal/workloads/workload.go (new shared helper)
• Corresponding _test.go files