Skip to content

feat(evals): add eval suite for setup-override-upstream skill#334

Merged
potiuk merged 1 commit into
apache:mainfrom
justinmclean:eval-setup-override-upstream
May 27, 2026
Merged

feat(evals): add eval suite for setup-override-upstream skill#334
potiuk merged 1 commit into
apache:mainfrom
justinmclean:eval-setup-override-upstream

Conversation

@justinmclean

Copy link
Copy Markdown
Member

Summary

15 cases across 4 steps covering the override-upstreaming workflow: pre-flight repo/drift checks, override selection dispatch, upstreamability classification, and PR confirmation gating.

Generated-by: Claude (Opus 4.7)

Type of change

  • Skill change (.claude/skills/<name>/) — eval fixtures updated below
  • Tool / bridge contract (tools/<system>/*.md)
  • Python package (tools/*/ with pyproject.toml)
  • Groovy reference impl
  • Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM)
  • Documentation (docs/, README.md, CONTRIBUTING.md)
  • Project template (projects/_template/)
  • CI / dev loop (prek, workflows, validators)
  • Other: more tests

Test plan

  • prek run --all-files passes
  • For Python packages touched: uv run pytest / ruff check / mypy passes
  • For Groovy bridges touched: command-line invocation tested end-to-end
  • For skill changes: eval suite passes for the affected skill
    (PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/<skill>/)
  • For skill behaviour changes: a new or updated eval fixture is included in this PR
    (a regression test for the bug fixed / the behaviour added — see CONTRIBUTING.md)
  • Other:

@justinmclean justinmclean self-assigned this May 27, 2026
15 cases across 4 steps covering the override-upstreaming workflow:
pre-flight repo/drift checks, override selection dispatch,
upstreamability classification, and PR confirmation gating.

Generated-by: Claude (Opus 4.7)
@potiuk potiuk force-pushed the eval-setup-override-upstream branch from de495d4 to 2015b29 Compare May 27, 2026 19:11

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — 15 cases across the four highest-signal decision points in
setup-override-upstream (pre-flight repo/drift, override pick, upstreamability
classification, PR confirm). All four step-configs resolve to real SKILL.md
headings. The two prompt-injection cases (override-pick and upstreamability)
correctly preserve real-state output rather than the injected answer. Runner
walks all 15 cases without parse errors.


This review was drafted by an AI-assisted tool and confirmed by an Apache Steward
maintainer. The maintainer approving this PR has read the findings and signed off.
If something feels off, please reply on the PR and a maintainer will follow up.

More on how Apache Steward handles maintainer review:
CONTRIBUTING.md.

@potiuk potiuk merged commit 248becb into apache:main May 27, 2026
15 checks passed
@justinmclean justinmclean deleted the eval-setup-override-upstream branch May 28, 2026 00:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants