[Tech Debt]: workload_details.status never transitions to 'failed' on orchestration error

[mrhillsman]

Category

Error handling gaps

Description

docs/audit-schema.md documents the workload_details.status transitions as pending → created → failed. In practice:

• RecordWorkload inserts the row with the schema default status='pending'.
• UpdateWorkloadStatus(workloadID, "created") runs only after all VMs for that workload succeed.
• If any VM fails (vm_failed event), the orchestrator returns early. The parent workload_details row stays pending forever.

So failed is documented but never written. A query like "show me which workloads have failed in the last day" returns nothing useful.

Severity

Medium — increases maintenance burden or risk of future bugs

Affected Files

• cmd/virtwork/main.go — runE error paths around VM creation and readiness
• internal/audit/audit.go — UpdateWorkloadStatus already accepts arbitrary status strings
• docs/audit-schema.md — documents the failed transition

Proposed Fix

Pick one:

Option A — write failed on the parent. In runE, when a VM creation or readiness check fails for a given workload, call UpdateWorkloadStatus(workloadID, "failed") for that workload before returning. Add a test that asserts the transition.

Option B — narrow the doc. Update docs/audit-schema.md to say the status is only pending or created; failures are visible per-VM via the events table (event_type=vm_failed).

Option A is more useful for operators who want a quick "which workloads failed" view; Option B is the smaller change.

Area

Audit system

Architecture Layer

Layer 4 - Orchestration (cmd, cleanup)