Track retried transient errors in eval results for diagnostics #435
Description
Context
Follow-up from #431 (execution status classification).
Request
Add errorRetries field to EvaluationResult to preserve transient errors that were successfully retried during provider invocation.
Why
When retries succeed, the transient errors are currently lost. For diagnostics and reliability monitoring, users need visibility into retry patterns (e.g., frequent rate limits, intermittent timeouts).
Design
interface EvaluationResult {
// ... existing fields
errorRetries?: readonly {
readonly message: string;
readonly attempt: number;
readonly timestamp: string;
}[];
}In the orchestrator retry loop, capture each caught error before retrying. If the retry succeeds, attach the captured errors to the final result.
Related
- Add explicit tooling/execution-failure status separate from model score #431 — execution status classification (prerequisite, merged)