Skip to content

docs: fix stale skill validator references#354

Closed
nightcityblade wants to merge 2 commits into
apache:mainfrom
nightcityblade:fix/issue-351
Closed

docs: fix stale skill validator references#354
nightcityblade wants to merge 2 commits into
apache:mainfrom
nightcityblade:fix/issue-351

Conversation

@nightcityblade

Copy link
Copy Markdown

Summary

Type of change

  • Skill change (.claude/skills/<name>/) — eval fixtures updated below
  • Tool / bridge contract (tools/<system>/*.md)
  • Python package (tools/*/ with pyproject.toml)
  • Groovy reference impl
  • Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM)
  • Documentation (docs/, README.md, CONTRIBUTING.md)
  • Project template (projects/_template/)
  • CI / dev loop (prek, workflows, validators)
  • Other:

Test plan

  • prek run --all-files passes
  • For Python packages touched: uv run pytest / ruff check / mypy passes
  • For Groovy bridges touched: command-line invocation tested end-to-end
  • For skill changes: eval suite passes for the affected skill
    (PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/<skill>/)
  • For skill behaviour changes: a new or updated eval fixture is included in this PR
    (a regression test for the bug fixed / the behaviour added — see CONTRIBUTING.md)
  • Other: git grep -nE 'skill-validator|skill-validate\\b' -- ':!tools/skill-and-tool-validator' | grep -v skill-and-tool || true
  • Other: attempted uv run --project tools/skill-and-tool-validator --group dev skill-and-tool-validate, but local uv is pinned to 0.10.7 and the repo requires >=0.11.8

RFC-AI-0004 compliance

  • HITL — any new mutation is gated on explicit user confirmation
  • Sandbox — no new unrestricted host access; network reach declared in the adapter
  • Vendor neutrality — placeholders (<PROJECT>, <tracker>, <upstream>, <security-list>) used in all skill / tool prose (the check-placeholders prek hook is the mechanical gate)
  • Conversational + correctable — agentic-override path documented if behaviour is adopter-tunable
  • Write-access discipline — no autonomous outbound messages; drafts only, sent on confirmation
  • Privacy LLM — private content does not reach a non-approved LLM; redactor invoked where needed

Linked issues

Closes #351

Notes for reviewers (optional)

The change is docs-only. I left the test-plan note about the uv version mismatch so the failed local validator invocation is explicit rather than silently omitted.

@justinmclean

justinmclean commented May 28, 2026

Copy link
Copy Markdown
Member

Thanks for your contribution, @nightcityblade - this looks good to approve/merge.

The PR satisfies #351:

  • stale-reference grep is clean
  • setup-override-upstream eval suite passes
  • skill-and-tool-validate runs successfully
  • diff is narrow and matches the issue scope

Thanks for your note on the issue with uv run - we'll get that fixed

@andreahlert andreahlert left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @nightcityblade! Thanks for adding this!

Just a few things:

Rename is correct and scope is clean (verified tools/skill-and-tool-validator/ exists on main, tools/skill-validator/ does not). Three blockers before merge:

  1. Acceptance criterion #1 not met. Issue #351 requires the grep to return zero hits. One stale reference still ships in:

    tools/skill-evals/evals/setup-override-upstream/step-6-pr-confirm/fixtures/case-1-shows-all-sections/report.md:25
    - [ ] Ran `skill-validate` — passes.
    

    The issue flagged this fixture explicitly as "needs judgement, not blind replace" and asked for the setup-override-upstream eval to be re-run after the edit. I checked expected.json for this case and it does not literally reference skill-validate, so updating the fixture prose should be safe, but the decision needs to be explicit, not silent.

  2. Grep command in the test plan is broken. As written:

    git grep -nE 'skill-validator|skill-validate\\b' ...
    

    In a bash single-quoted string \\b is the literal two chars \b, which in ERE matches an escaped backslash followed by b, not a word boundary. The issue's original command used \b (single backslash = ERE word boundary). The version pasted here silently returns zero hits, which is why the missed fixture above was not caught. If that is just markdown escaping when pasting, the grep that was actually run was the working one — please confirm — but the command as recorded in this PR does not run as the author claims.

  3. Validator not actually executed. Test plan notes uv is pinned to 0.10.7 vs required >=0.11.8, which is fine to disclose, but the diff also touches prose in meta-and-quality-tooling.md (tool name and CLI description, not just commands). Running skill-and-tool-validate once on a compatible uv would catch any link/frontmatter regressions in that prose change. Please run it before merge.

Once the fixture is updated (or its non-update is justified in a comment) and the eval re-run, this is ready to merge.

Comment thread tools/spec-loop/AGENTS.md
```bash
# Validate skill definitions (frontmatter, links, placeholders)
uv run --project tools/skill-validator --group dev skill-validate
uv run --project tools/skill-and-tool-validator --group dev skill-and-tool-validate

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This same rename is missing from tools/skill-evals/evals/setup-override-upstream/step-6-pr-confirm/fixtures/case-1-shows-all-sections/report.md:25 (- [ ] Ran \skill-validate` — passes.). Issue #351 acceptance criterion #1 (git grep` returns no hits) is not met until that fixture is updated or its non-update is justified. The issue called it out explicitly under "Two spots need judgement, not blind replace."

source: >
README.md § Skill families (utilities) and AGENTS.md § Reusable skills.
Implemented by tools/skill-validator/, tools/skill-evals/,
Implemented by tools/skill-and-tool-validator/, tools/skill-evals/,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prose change here is correct. Reminder from the issue: after updating the setup-override-upstream fixture, re-run uv run --project tools/skill-evals skill-eval tools/skill-evals/evals/setup-override-upstream/ and confirm expected.json still matches. expected.json for case-1-shows-all-sections does not literally contain skill-validate, so the update is most likely safe, but please verify.

@andreahlert andreahlert requested a review from justinmclean May 28, 2026 07:49
@andreahlert andreahlert added documentation Improvements or additions to documentation family:docs Docs, MISSION.md, READMEs family:tools tools/* mode:platform Substrate / infra — not a mode (sandbox, CI, validators) labels May 28, 2026
@nightcityblade

Copy link
Copy Markdown
Author

Thanks for the careful review — I pushed 8051647 to address the missed fixture reference, so the setup-override-upstream eval now uses skill-and-tool-validate consistently. I also reran PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/setup-override-upstream locally after the change and it passed.

On the grep note: the command I actually ran locally used a single \b; the doubled form in the PR text was just a bad paste in the write-up, so thanks for catching that.

I still have one gap on the validator run itself: this checkout has uv 0.10.7, while the repo now requires >=0.11.8, so uv run --project tools/skill-and-tool-validator --group dev skill-and-tool-validate exits before execution with the version check. If you want, I can refresh that once I have a newer uv available, but I didn't want to fake a successful run in the meantime.

@potiuk

potiuk commented May 28, 2026

Copy link
Copy Markdown
Member

Hi @nightcityblade — thanks very much for catching this and putting the fix together. I'm closing this in favour of #352, which @MD-Mushfiqur123 opened about an hour and a half earlier with the same scope. Both PRs touched the exact same 9 files and made essentially the same find-and-replace from skill-validator / skill-validate to skill-and-tool-validator / skill-and-tool-validate — yours was solid; it just happened to land in the queue second.

Your work is real and helpful — we'd be glad to see more contributions from you. If you'd like a heads-up on what's likely to be picked up next, the open issues with good first issue are a good place to start, and the area:* labels can help you find something close to your interests.

Thanks again, and apologies for the duplicate-effort frustration — that's on us, not on you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation family:docs Docs, MISSION.md, READMEs family:tools tools/* mode:platform Substrate / infra — not a mode (sandbox, CI, validators)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Update stale skill-validator / skill-validate references to the renamed skill-and-tool-validator

4 participants