Skip to content

Surface ticket-vs-PR-description divergence in /review-pr #567

Description

@williamthorsen

Problem

/review-pr evaluates the implementation against both the ticket and the PR description as separate specification sources, rendering each as a row in ## Specification compliance. What it does not surface is divergence between the two sources — cases where the ticket and the PR description make different or contradictory claims independent of the implementation. The most consequential variant is semantic contradiction (e.g., ticket says "block requests over 10MB"; PR description says "100MB") — the implementation can pass against the source it followed and still leave the contradiction unaddressed.

Context

Follow-up to #553, which introduced the two-source compliance model. The spec-vs-spec check was deferred there because it raises classification and shape questions that the compliance-against-implementation flow does not.

Related files:

  • packages/agents/content/skills/review-branch/SKILL.md — canonical home of the shared review process and output template
  • packages/agents/content/skills/review-pr/SKILL.md — thin entry skill; carries a deferred-work note that comes out as part of this change
  • packages/agents/content/skills/_data/next-steps-after-review.md — interactive next-steps generator that currently triggers on ticket-compliance gaps only

Proposed solution

Add specification consistency as an assessment dimension modeled on assess-ticket's drift gradient — a single verdict plus evidence bullets, not a list of findings.

Output shape (rendered as a new top-level ## Specification consistency section immediately after ## Specification compliance):

  • 🟢 none — sources align semantically; paraphrase is not divergence
  • 🟠 partial — minor reframing, scope-trim, or omissions worth reader attention
  • 🔴 severe — direct contradictions or scope conflicts

Evidence bullets describe the specific divergences (or note "PR description defers to ticket — no independent assertions to compare" when applicable).

Rendering condition: render when spec_sources.length >= 2, omit otherwise. Currently only /review-pr produces ≥2 sources, so the section appears only there.

The consistency check is independent of the F/W/T/R/S finding scheme. If a divergence also surfaces something author-actionable (e.g., the implementation matches neither source), that flows through the existing finding scheme — the consistency section's job is to flag the source-pair mismatch, not to duplicate compliance findings.

The interactive next-steps generator gains a trigger on partial / severe consistency, with options to update the ticket, update the PR description, or accept the divergence.

Acceptance criteria

  • review-branch/SKILL.md defines a ## Specification consistency output section with the three-tier verdict scale and evidence-bullet format, rendered when spec_sources.length >= 2 and omitted otherwise.
  • The verdict scale uses 🟢 none / 🟠 partial / 🔴 severe to match assess-ticket's concern-scale convention.
  • The skill text explicitly answers the paraphrase tolerance, defer-to-ticket, and classification questions in line with the design (paraphrase ≠ divergence; defer-to-ticket renders none with an evidence bullet; verdict is independent of F/W/T/R/S).
  • next-steps-after-review.md triggers a sub-block on partial or severe consistency with appropriate options.
  • review-pr/SKILL.md removes the "deferred (tracked separately)" note about spec-vs-spec divergence now that it is part of the canonical output.
  • No existing per-source compliance rendering changes.

Metadata

Metadata

Labels

featureAdded or improved external functionalityscope:agents

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions