Skip to content

feat(evals): add eval suite for setup-shared-config-sync skill#331

Merged
potiuk merged 1 commit into
apache:mainfrom
justinmclean:eval-setup-shared-config-sync
May 27, 2026
Merged

feat(evals): add eval suite for setup-shared-config-sync skill#331
potiuk merged 1 commit into
apache:mainfrom
justinmclean:eval-setup-shared-config-sync

Conversation

@justinmclean

Copy link
Copy Markdown
Member

Summary

11 cases across 2 steps covering the action-path decision (in-sync, push-only, commit-then-push, pull-then-commit-then-push, not-a-git-repo, lock-held, injection resistance) and commit-message drafting (update existing script, add new config file, multi-file commit, injection in diff). step-3-decide-action cases are auto-comparable in --cli mode; step-5-draft-commit uses structural has_* flags for manual review. Updates tools/skill-evals/README.md suite count from 18 to 19.

Generated-by: Claude (Opus 4.7)

Type of change

  • Skill change (.claude/skills/<name>/) — eval fixtures updated below
  • Tool / bridge contract (tools/<system>/*.md)
  • Python package (tools/*/ with pyproject.toml)
  • Groovy reference impl
  • Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM)
  • Documentation (docs/, README.md, CONTRIBUTING.md)
  • Project template (projects/_template/)
  • CI / dev loop (prek, workflows, validators)
  • Other: more tests

Test plan

  • prek run --all-files passes
  • For Python packages touched: uv run pytest / ruff check / mypy passes
  • For Groovy bridges touched: command-line invocation tested end-to-end
  • For skill changes: eval suite passes for the affected skill
    (PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/<skill>/)
  • For skill behaviour changes: a new or updated eval fixture is included in this PR
    (a regression test for the bug fixed / the behaviour added — see CONTRIBUTING.md)
  • Other:

@justinmclean justinmclean self-assigned this May 27, 2026
@potiuk potiuk force-pushed the eval-setup-shared-config-sync branch from c3a8f80 to 70d9d9d Compare May 27, 2026 19:04

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — 11 cases across the two highest-signal steps in
setup-shared-config-sync (decide-action and draft-commit). Both
step-configs resolve to a real SKILL.md heading. Two prompt-injection
cases — one in the git-status output, one in a diff line — both
correctly preserve real-state output. Runner walks all 11 cases without
parse errors.

Smaller observations

  • Both step-configs point to the same ## Walk-through heading (the
    only top-level heading covering the action flow), so the runner
    extracts the same system-prompt content for both step evals. This
    works — the model sees the full walk-through and the case data
    differentiates which step is being exercised — but the precision is
    lower than in the sibling eval suites (#332/#333/#334/#335), which
    have distinct headings per step. If the skill's Walk-through ever
    gets refactored into ### Step N — … subheadings, the step-configs
    should be retargeted accordingly. Not blocking.

This review was drafted by an AI-assisted tool and confirmed by an Apache Steward
maintainer. The maintainer approving this PR has read the findings and signed off.
If something feels off, please reply on the PR and a maintainer will follow up.

More on how Apache Steward handles maintainer review:
CONTRIBUTING.md.

11 cases across 2 steps covering the action-path decision (in-sync,
push-only, commit-then-push, pull-then-commit-then-push, not-a-git-repo,
lock-held, injection resistance) and commit-message drafting (update
existing script, add new config file, multi-file commit, injection in
diff). step-3-decide-action cases are auto-comparable in --cli mode;
step-5-draft-commit uses structural has_* flags for manual review.
Updates tools/skill-evals/README.md suite count from 18 to 19.

Generated-by: Claude (Opus 4.7)
@potiuk potiuk force-pushed the eval-setup-shared-config-sync branch from 70d9d9d to b33ba47 Compare May 27, 2026 22:35
@potiuk potiuk merged commit 261adb4 into apache:main May 27, 2026
15 checks passed
@justinmclean justinmclean deleted the eval-setup-shared-config-sync branch May 28, 2026 00:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants