feat(evals): add eval suite for setup-steward skill#335
Conversation
12 cases across two suites: - step-conventions-detect (7): Pattern A/B/C/D.1/D.2, ambiguous (both dirs exist as regular dirs with independent content), and prompt-injection resistance - step-verify-drift (5): clean, method/URL mismatch, ref mismatch, svn-zip SHA-512 mismatch (security-flagged), local lock missing Both suites are fully auto-comparable in --cli mode. Validates the two highest-signal decision points in setup-steward: the skills-dir convention detection algorithm and the committed-vs-local lock drift check that every framework skill runs at the top of its invocation. Generated-by: Claude (Opus 4.7)
potiuk
force-pushed
the
eval-setup-steward
branch
from
fedc4d2 to
6358e11
Compare
potiuk
left a comment
potiuk
left a comment
There was a problem hiding this comment.
LGTM — two well-formed eval suites covering the two highest-signal decision points in
setup-steward. Step-configs resolve to real skill headings (conventions.md §
Detection algorithm, verify.md § 3. Drift between committed and local locks).
The prompt-injection case correctly preserves real-state output with
injection_flagged: true rather than the injected answer. Runner walks all 12 cases
without parse errors.
This review was drafted by an AI-assisted tool and confirmed by an Apache Steward
maintainer. The maintainer approving this PR has read the findings and signed off.
If something feels off, please reply on the PR and a maintainer will follow up.More on how Apache Steward handles maintainer review:
CONTRIBUTING.md.
2 participants
Summary
12 cases across two suites:
Both suites are fully auto-comparable in --cli mode. Validates the two highest-signal decision points in setup-steward: the skills-dir convention detection algorithm and the committed-vs-local lock drift check that every framework skill runs at the top of its invocation.
Generated-by: Claude (Opus 4.7)
Type of change
.claude/skills/<name>/) — eval fixtures updated belowtools/<system>/*.md)tools/*/withpyproject.toml)docs/,README.md,CONTRIBUTING.md)projects/_template/)prek, workflows, validators)Test plan
prek run --all-filespassesuv run pytest/ruff check/mypypasses(
PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/<skill>/)(a regression test for the bug fixed / the behaviour added — see CONTRIBUTING.md)