Generic skill-failure logging (JSONL with project_slug, skill, timestamp)

[williamthorsen]

Problem

Several skills already prescribe a recovery protocol that surfaces information about failures (e.g., the bisection trace in update-jira-ticket's INVALID_INPUT recovery), but there is no shared mechanism for persisting those failures as structured data. Without persisted records, decisions to escalate (e.g., "should we add a deterministic sanitiser to this skill?") rest on vibes rather than evidence — and historically that means escalation never happens, because the data isn't there to motivate it.

The motivating use case is #467, which ships its skill body-only on the explicit bet that observed failure rate will tell us whether to add a sanitiser later. To make "later" possible, the skill writes failure records to an interim ad-hoc JSONL location (~/ai-artifacts/skill-failures/update-jira-ticket.jsonl). That works for one skill but doesn't scale: every skill that wants to do this would invent its own location and field set, leading to inconsistent paths, drift, and no shared review tooling.

A generic logging mechanism — even a minimal one — would let any skill record a failure consistently, with a small, predictable surface for future review.

Context

• The pattern emerged from the design discussion on Eliminate input failures when updating Jira issues via MCP #467; see that ticket for the originating use case.
• Ad-hoc location currently used: ~/ai-artifacts/skill-failures/update-jira-ticket.jsonl. This is the kind of thing that should converge on a project convention rather than diverge per skill.
• Several skills could adopt this immediately if it existed (any skill with a recovery/backstop protocol that wants to inform future hardening decisions).

Solution

Define and document a generic skill-failure logging mechanism, and provide a tiny helper any skill can call from its recovery protocol.

Minimum field set

Each record is one JSON object per line, with at least:

Field	Type	Description
timestamp	string	ISO 8601 UTC, e.g., 2026-04-28T03:15:32Z
skill	string	Skill name, e.g., update-jira-ticket
project_slug	string	Project the failure occurred in (repo basename or convention)
payload	object	Open-ended skill-specific fields (failing fragment, error message, etc.)

Storage

• One canonical root, e.g., ~/ai-artifacts/skill-failures/.
• Either one file per skill (<skill>.jsonl) or one global file (failures.jsonl) with skill as the discriminator. Pick one convention and document it.
• Append-only; writers must not truncate or reorder.
• Create the directory and file lazily on first write.

Helper

A small helper that any skill's recovery protocol can invoke without boilerplate. Possible shapes (decide during design):

• A short shell function shipped in packages/agents/scripts/functions/ (joins the existing colors.sh pattern).
• A tiny Python script in the agents content tree (mirrors any other Python helpers we add).
• A codeassembly-agents CLI subcommand (heaviest lift; only worth it if multi-skill adoption is high).

Out of scope (for this ticket)

• Log rotation.
• Querying tooling, dashboards, summarisation.
• Aggregating failures across machines.

These are all reasonable later additions; this ticket is just about establishing the persistence convention and a one-line writer.

Acceptance criteria

• A documented convention exists for the storage location and JSONL field set, including the four required fields (timestamp, skill, project_slug, payload).
• A helper (shell function, script, or CLI subcommand) is shipped that any skill's recovery protocol can call to write a record without per-skill boilerplate.
• At least one existing skill (update-jira-ticket from Eliminate input failures when updating Jira issues via MCP #467 once landed) is migrated from its interim ad-hoc location to the generic mechanism.
• The convention and helper are documented somewhere discoverable (e.g., a _data snippet referenced from skills, or the agents package README).
• New behavior is covered by tests where the helper is testable code (a shell function or script).