feat: classify driver-exit (0-turn) failures separately from agent-logic failures in health metrics

docs/adr/47250-classify-driver-exit-vs-agent-logic-failures.md

@@ -0,0 +1,50 @@
+# ADR-47250: Classify Driver-Exit (0-Turn) Failures Separately from Agent-Logic Failures
+
+**Date**: 2026-07-22
+**Status**: Draft
+**Deciders**: Unknown (copilot-swe-agent, pelikhan)
+
+---
+
+### Context
+
+Fleet health dashboards aggregate all failed workflow runs into a single failure count. Some failures occur before the agent ever runs — the CLI wrapper or a pre/post-agent infrastructure step exits non-zero with zero agent turns recorded. These infra-level failures are indistinguishable from genuine agent-logic failures (where the agent ran but the run still concluded as failure), causing health dashboards to overstate fleet ill-health and making it harder to detect actual agent regressions versus transient infrastructure flakiness.
+
+The existing `WorkflowRun` data model already tracks `Turns` (the number of agent conversation turns), which provides a reliable proxy: zero turns means the agent never ran, implying the failure originated in the driver layer.
+
+### Decision
+
+We will add a `driver_exit` vs `agent_logic` failure classifier throughout the health-metrics and logs-report surfaces. The classification rule is: a failed run with `Turns == 0` is a `driver_exit` failure; a failed run with `Turns > 0` is an `agent_logic` failure. This rule is implemented as `isDriverExitFailure(run WorkflowRun) bool` in `pkg/cli/logs_models.go` and propagated to `WorkflowHealth.DriverExitCount`, `WorkflowHealth.AgentLogicFailureCount`, `LogsSummary.TotalDriverExitFailures`, `LogsSummary.TotalAgentLogicFailures`, and the per-run `RunData.FailureKind` field. Operators can filter on `failure_kind` to separate infra flakiness from agent regressions without manually cross-referencing turn counts.
+
+### Alternatives Considered
+
+#### Alternative 1: Operator-Side Filtering (No Schema Change)
+
+Operators could filter `turns == 0` themselves when consuming the existing JSON output, without adding any new fields to the data model. This avoids expanding the public schema.
+
+**Why not chosen**: Every consumer would need to re-derive the classification independently, creating duplicated logic and risk of inconsistency. Embedding the classification in the output makes the distinction first-class and self-documenting for all consumers, including dashboards and automated alerts.
+
+#### Alternative 2: Richer Multi-Category Failure Taxonomy
+
+Instead of a binary `driver_exit`/`agent_logic` split keyed solely on turn count, introduce a broader `failure_category` enum (e.g., `infra_pre`, `infra_post`, `agent_logic`, `agent_timeout`) using additional signals such as log markers, job step names, or exit codes.
+
+**Why not chosen**: Additional signals are not consistently available across all run types and would require deeper changes to the data collection pipeline. The turn-count heuristic is immediately available, simple to verify in tests, and addresses the primary dashboard noise problem without waiting for richer instrumentation.
+
+### Consequences
+
+#### Positive
+- Health dashboards no longer conflate infrastructure flakiness with agent regressions; fleet ill-health metrics become more actionable.
+- The classification heuristic is simple (`turns == 0`) and fully testable without external dependencies, reducing maintenance burden.
+- `failure_kind` is an additive, opt-in field (`omitempty`) so existing consumers that ignore it are unaffected.
+
+#### Negative
+- The zero-turn heuristic is a proxy, not a guarantee: a run that fails at the agent's very first action (before completing a turn) would still report `Turns == 0` and be misclassified as `driver_exit` even if the agent did begin executing.
+- `failure_kind`, `driver_exit_count`, and `agent_logic_failure_count` are new semi-stable JSON schema fields. Once dashboards and tooling build on them, changing the heuristic or renaming the fields will require a coordinated migration.
+
+#### Neutral
+- `WorkflowHealth` gains two new exported fields that appear in JSON output; consumers doing strict schema validation may need minor updates.
+- The implementation spans three files (`logs_models.go`, `health_metrics.go`, `logs_report.go`), keeping the classifier logic centralized in `logs_models.go` and consumed by the other two.
+
+---
+
+*ADR created by [adr-writer agent]. Review and finalize before changing status from Draft to Accepted.*

pkg/cli/health_metrics.go

@@ -14,19 +14,21 @@ var healthMetricsLog = logger.New("cli:health_metrics")
 
 // WorkflowHealth represents health metrics for a single workflow
 type WorkflowHealth struct {
-	WorkflowName  string        `json:"workflow_name" console:"header:Workflow"`
-	TotalRuns     int           `json:"total_runs" console:"-"`
-	SuccessCount  int           `json:"success_count" console:"-"`
-	FailureCount  int           `json:"failure_count" console:"-"`
-	SuccessRate   float64       `json:"success_rate" console:"-"`
-	DisplayRate   string        `json:"-" console:"header:Success Rate"`
-	Trend         string        `json:"trend" console:"header:Trend"`
-	AvgDuration   time.Duration `json:"avg_duration" console:"-"`
-	DisplayDur    string        `json:"-" console:"header:Avg Duration"`
-	TotalTokens   int           `json:"total_tokens" console:"-"`
-	AvgTokens     int           `json:"avg_tokens" console:"-"`
-	DisplayTokens string        `json:"-" console:"header:Avg Tokens"`
-	BelowThresh   bool          `json:"below_threshold" console:"-"`
+	WorkflowName           string        `json:"workflow_name" console:"header:Workflow"`
+	TotalRuns              int           `json:"total_runs" console:"-"`
+	SuccessCount           int           `json:"success_count" console:"-"`
+	FailureCount           int           `json:"failure_count" console:"-"`
+	DriverExitCount        int           `json:"driver_exit_count" console:"-"`
+	AgentLogicFailureCount int           `json:"agent_logic_failure_count" console:"-"`
+	SuccessRate            float64       `json:"success_rate" console:"-"`
+	DisplayRate            string        `json:"-" console:"header:Success Rate"`
+	Trend                  string        `json:"trend" console:"header:Trend"`
+	AvgDuration            time.Duration `json:"avg_duration" console:"-"`
+	DisplayDur             string        `json:"-" console:"header:Avg Duration"`
+	TotalTokens            int           `json:"total_tokens" console:"-"`
+	AvgTokens              int           `json:"avg_tokens" console:"-"`
+	DisplayTokens          string        `json:"-" console:"header:Avg Tokens"`
+	BelowThresh            bool          `json:"below_threshold" console:"-"`
 }
 
 // HealthSummary represents aggregated health metrics across all workflows
@@ -78,6 +80,8 @@ func CalculateWorkflowHealth(workflowName string, runs []WorkflowRun, threshold
 	// Accumulate success/failure counts and numerical metrics.
 	successCount := 0
 	failureCount := 0
+	driverExitCount := 0
+	agentLogicFailureCount := 0
 	var durationStats, tokenStats stats.StatVar
 	var totalTokens int
 
@@ -86,6 +90,13 @@ func CalculateWorkflowHealth(workflowName string, runs []WorkflowRun, threshold
 			successCount++
 		} else if isFailureConclusion(run.Conclusion) {
 			failureCount++
+			if isDriverExitFailure(run) {
+				driverExitCount++
+			} else if run.TurnsAvailable || run.Turns > 0 {
+				// Only count as agent-logic when we have reliable turn data.
+				// Runs without artifact logs (TurnsAvailable=false, Turns=0) are left unclassified.
+				agentLogicFailureCount++
+			}
 		}
 		totalTokens += run.TokenUsage
 		durationStats.Add(float64(run.Duration))
@@ -109,19 +120,21 @@ func CalculateWorkflowHealth(workflowName string, runs []WorkflowRun, threshold
 	belowThreshold := successRate < threshold
 
 	health := WorkflowHealth{
-		WorkflowName:  workflowName,
-		TotalRuns:     totalRuns,
-		SuccessCount:  successCount,
-		FailureCount:  failureCount,
-		SuccessRate:   successRate,
-		DisplayRate:   displayRate,
-		Trend:         trend.String(),
-		AvgDuration:   avgDuration,
-		DisplayDur:    displayDur,
-		TotalTokens:   totalTokens,
-		AvgTokens:     avgTokens,
-		DisplayTokens: displayTokens,
-		BelowThresh:   belowThreshold,
+		WorkflowName:           workflowName,
+		TotalRuns:              totalRuns,
+		SuccessCount:           successCount,
+		FailureCount:           failureCount,
+		DriverExitCount:        driverExitCount,
+		AgentLogicFailureCount: agentLogicFailureCount,
+		SuccessRate:            successRate,
+		DisplayRate:            displayRate,
+		Trend:                  trend.String(),
+		AvgDuration:            avgDuration,
+		DisplayDur:             displayDur,
+		TotalTokens:            totalTokens,
+		AvgTokens:              avgTokens,
+		DisplayTokens:          displayTokens,
+		BelowThresh:            belowThreshold,
 	}
 
 	healthMetricsLog.Printf("Health calculated: workflow=%s, successRate=%.2f%%, trend=%s", workflowName, successRate, trend.String())

pkg/cli/health_metrics_test.go

@@ -257,3 +257,102 @@ func TestFormatTokens(t *testing.T) {
 		})
 	}
 }
+
+func TestCalculateWorkflowHealthDriverExitClassification(t *testing.T) {
+	runs := []WorkflowRun{
+		{Conclusion: "success", Duration: 2 * time.Minute, Turns: 3, TurnsAvailable: true},
+		// driver-exit: failed with zero turns (TurnsAvailable confirms the count is real)
+		{Conclusion: "failure", Duration: 1 * time.Minute, Turns: 0, TurnsAvailable: true},
+		{Conclusion: "failure", Duration: 1 * time.Minute, Turns: 0, TurnsAvailable: true},
+		// agent-logic: failed but agent ran (turns > 0)
+		{Conclusion: "failure", Duration: 2 * time.Minute, Turns: 5, TurnsAvailable: true},
+	}
+
+	health := CalculateWorkflowHealth("test-workflow", runs, 80.0)
+
+	assert.Equal(t, 4, health.TotalRuns, "total runs should be 4")
+	assert.Equal(t, 1, health.SuccessCount, "success count should be 1")
+	assert.Equal(t, 3, health.FailureCount, "failure count should be 3")
+	assert.Equal(t, 2, health.DriverExitCount, "driver-exit count should be 2 (zero-turn failures)")
+	assert.Equal(t, 1, health.AgentLogicFailureCount, "agent-logic count should be 1 (non-zero-turn failure)")
+}
+
+func TestCalculateWorkflowHealthDriverExitCountZeroWhenNoFailures(t *testing.T) {
+	runs := []WorkflowRun{
+		{Conclusion: "success", Duration: 2 * time.Minute, Turns: 3},
+		{Conclusion: "success", Duration: 3 * time.Minute, Turns: 4},
+	}
+
+	health := CalculateWorkflowHealth("test-workflow", runs, 80.0)
+
+	assert.Equal(t, 0, health.DriverExitCount, "driver-exit count should be zero when all runs succeed")
+	assert.Equal(t, 0, health.AgentLogicFailureCount, "agent-logic count should be zero when all runs succeed")
+}
+
+func TestCalculateWorkflowHealthTurnsUnavailableSkipsClassification(t *testing.T) {
+	// Simulate the gh aw health path: GitHub API metadata only, no artifact logs,
+	// so TurnsAvailable is false for every run. Classification should be skipped and
+	// both DriverExitCount and AgentLogicFailureCount must stay at zero.
+	runs := []WorkflowRun{
+		{Conclusion: "success", Duration: 2 * time.Minute},
+		{Conclusion: "failure", Duration: 1 * time.Minute, Turns: 0, TurnsAvailable: false},
+		{Conclusion: "failure", Duration: 1 * time.Minute, Turns: 0, TurnsAvailable: false},
+	}
+
+	health := CalculateWorkflowHealth("test-workflow", runs, 80.0)
+
+	assert.Equal(t, 2, health.FailureCount, "failure count should still be 2")
+	assert.Equal(t, 0, health.DriverExitCount, "driver-exit count should be zero when TurnsAvailable=false")
+	assert.Equal(t, 0, health.AgentLogicFailureCount, "agent-logic count should be zero when TurnsAvailable=false")
+}
+
+func TestIsDriverExitFailure(t *testing.T) {
+	tests := []struct {
+		name     string
+		run      WorkflowRun
+		expected bool
+	}{
+		{
+			name:     "failure with zero turns is driver-exit",
+			run:      WorkflowRun{Conclusion: "failure", Turns: 0, TurnsAvailable: true},
+			expected: true,
+		},
+		{
+			name:     "timed_out with zero turns is driver-exit",
+			run:      WorkflowRun{Conclusion: "timed_out", Turns: 0, TurnsAvailable: true},
+			expected: true,
+		},
+		{
+			name:     "cancelled with zero turns is driver-exit",
+			run:      WorkflowRun{Conclusion: "cancelled", Turns: 0, TurnsAvailable: true},
+			expected: true,
+		},
+		{
+			name:     "failure with non-zero turns is agent-logic (not driver-exit)",
+			run:      WorkflowRun{Conclusion: "failure", Turns: 3, TurnsAvailable: true},
+			expected: false,
+		},
+		{
+			name:     "failure with zero turns but TurnsAvailable=false is unclassified",
+			run:      WorkflowRun{Conclusion: "failure", Turns: 0, TurnsAvailable: false},
+			expected: false,
+		},
+		{
+			name:     "success with zero turns is not driver-exit",
+			run:      WorkflowRun{Conclusion: "success", Turns: 0},
+			expected: false,
+		},
+		{
+			name:     "success with non-zero turns is not driver-exit",
+			run:      WorkflowRun{Conclusion: "success", Turns: 5},
+			expected: false,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			result := isDriverExitFailure(tt.run)
+			assert.Equal(t, tt.expected, result)
+		})
+	}
+}

pkg/cli/logs_models.go

@@ -73,6 +73,7 @@ type WorkflowRun struct {
 	ActionMinutes       float64 // Billable Actions minutes estimated from wall-clock time
 	TokenUsage          int
 	Turns               int
+	TurnsAvailable      bool // True when turn count was successfully read from artifact logs
 	ErrorCount          int
 	WarningCount        int
 	MissingToolCount    int
@@ -349,3 +350,12 @@ func isFailureConclusion(conclusion string) bool {
 	}
 	return isFailure
 }
+
+// isDriverExitFailure returns true when a failed run shows no agent turns, which
+// indicates the CLI wrapper or a pre/post-agent infrastructure step exited non-zero
+// before the agent had a chance to run.  Runs with Turns > 0 are classified as
+// agent-logic failures instead because the agent did execute.
+// TurnsAvailable must be true to confirm the zero is real rather than missing data.
+func isDriverExitFailure(run WorkflowRun) bool {
+	return isFailureConclusion(run.Conclusion) && run.TurnsAvailable && run.Turns == 0
+}