Skip to content

Add ticket-evaluation guidance to /design-and-plan #460

Description

@williamthorsen

Problem

Agents read tickets as authoritative directives rather than as the requests, reports, or suggestions they actually are. The default behavior is "the ticket says X, deliver X" — which means a ticket that's poorly framed, scoped too narrowly, or that proposes an unsound fix gets implemented as written. The agent's evaluation, audit, and judgment-calibration only kick in if the ticket happens to be wrong in a way the user catches mid-design.

Concrete instance from afg-devtools #379: the ticket described a CI flake in a single test file and proposed a minimal three-line fix. An audit during design surfaced two more files in the same package with the same defect class. The agent's initial framing kept proposing "fix this file in this PR, file follow-up tickets for the rest" because the ticket only named one file. Only after the user pushed back ("Tickets are not the 10 commandments… we should not be constrained by the ticket") did the agent broaden scope to the underlying defect class.

Context

Ticket-evaluation guidance was originally proposed for ~/.agents/AGENTS.md. On review, that placement was rejected for two reasons:

  • Token economics: AGENTS.md loads in every conversation. A ~200-word passage that only fires during ticket-driven design pays a global token cost for a localized benefit.
  • Closer fit downstream: the failure mode happens inside the design loop, and /design-and-plan is exactly that loop. It is the dominant entry point for ticket-driven design and already contains a related but narrower check (Phase 1 invokes /assess-ticket in drift mode for factual anchoring).

The new guidance is orthogonal to drift — it covers scope correctness, solution soundness, and title accuracy, which /assess-ticket does not currently address in any mode.

A separate follow-up ticket will propose extending /assess-ticket itself with the same evaluation criteria, since users invoke that skill directly for ticket triage. To prevent the four-facet wording from drifting between two consumers, the criteria are factored into a shared _data file from the start, following the existing pattern (_data/ticket-source-resolution.md, _data/complexity-classification.md, etc.).

Solution

  1. Create packages/agents/content/skills/_data/ticket-evaluation.md containing the four-facet evaluation criteria and the directive to raise divergences before designing:

    # Ticket evaluation
    
    A ticket is a request, report, or suggestion — not a contract. The author may have framed it poorly, missed the actual problem, scoped it too narrowly, proposed an unsound fix, or filed work that turns out to be inadvisable. Before designing to a ticket, evaluate it on its merits:
    
    - **Is the problem real?** Reproduce or verify the underlying observation, not just the framing.
    - **Is the scope right?** Look for related instances of the same defect class, neighboring code paths, or assumptions that don't hold beyond the ticket's frame.
    - **Is the proposed solution sound?** A ticket may include a recommended fix that solves the symptom but not the cause, or that fixes the cause less well than an alternative.
    - **Is the title accurate?** "Flaky test in X" may be a hygiene anti-pattern that exists in three places. Reframe titles when the underlying truth is broader or different.
    
    When evaluation surfaces a divergence from the ticket as written, raise the observation to the user before designing — but lead with the substantive finding, not the ticket. The ticket's job was to start the conversation; once it has started, the codebase reality and the user's judgment govern, not the original wording.
  2. Modify packages/agents/content/skills/design-and-plan/SKILL.md Phase 2 to add a new step between "Explore project context" and "Ask clarifying questions":

    2. **Evaluate the ticket on its merits** — apply the criteria in [ticket evaluation](../_data/ticket-evaluation.md). When evaluation surfaces a divergence from the ticket as written, raise it to the user before forming questions or designing.

    Renumber the existing "Ask clarifying questions" step to 3.

Compatibility with existing rules

Compatible with the no-derive-by-construction feedback rule: that rule prohibits expanding scope into new architectural patterns; this guidance expands scope to more instances of the same defect class. Different axes; no conflict.

Complementary to /assess-ticket Phase 1 invocation: drift mode catches "files moved or APIs renamed"; ticket evaluation catches "scope is wrong" or "proposed fix is unsound." No overlap in current call sites.

Acceptance criteria

  • packages/agents/content/skills/_data/ticket-evaluation.md exists, with the four-facet criteria and the "raise divergences" directive.
  • packages/agents/content/skills/design-and-plan/SKILL.md Phase 2 includes a new "Evaluate the ticket on its merits" step that references the _data file.
  • The new step is sequenced between "Explore project context" and "Ask clarifying questions"; existing step numbering is updated accordingly.
  • No content is added to ~/.agents/AGENTS.md (the original proposal target — explicitly out of scope).

Metadata

Metadata

Labels

featureAdded or improved external functionalityscope:agents

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions