Skip to content

feat(validator): add docs/modes.md consistency check (#11)#652

Merged
potiuk merged 1 commit into
apache:mainfrom
justinmclean:modes-doc-consistency-check
Jun 30, 2026
Merged

feat(validator): add docs/modes.md consistency check (#11)#652
potiuk merged 1 commit into
apache:mainfrom
justinmclean:modes-doc-consistency-check

Conversation

@justinmclean

Copy link
Copy Markdown
Member

Summary

Add validate_modes_doc_consistency (SOFT, category modes-doc-consistency) to the skill-and-tool-validator. The check compares docs/modes.md against live skills/*/SKILL.md frontmatter across four dimensions:

  1. Listed skill missing from disk — row exists in modes.md but skills// does not.
  2. Mode mismatch — a skill's mode: frontmatter differs from the section it appears in (e.g. listed under Triage but mode: Mentoring).
  3. Skill count mismatch — the integer in "Modes at a glance" Skill count column does not equal the actual row count in that mode's section.
  4. Unlisted skill — a live skill has a valid mode: frontmatter value but is absent from the corresponding docs/modes.md section.

On the live tree the check surfaces two real gaps:

  • reviewer-routing has mode: Triage but is absent from docs/modes.md
  • good-first-issue-sweep has mode: Mentoring but is absent from docs/modes.md

Ships 19 tests in TestParseModesDocs + TestValidateModeDocConsistency; all 283 validator tests pass.

Generated-by: Claude (Opus 4.7)

Type of change

  • Skill change (.claude/skills/<name>/) — eval fixtures updated below
  • Tool / bridge contract (tools/<system>/*.md)
  • Python package (tools/*/ with pyproject.toml)
  • Groovy reference impl
  • Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM)
  • Documentation (docs/, README.md, CONTRIBUTING.md)
  • Project template (projects/_template/)
  • CI / dev loop (prek, workflows, validators)
  • Other:

Test plan

  • prek run --all-files passes
  • For Python packages touched: uv run pytest / ruff check / mypy passes
  • For Groovy bridges touched: command-line invocation tested end-to-end
  • For skill changes: eval suite passes for the affected skill
    (PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/<skill>/)
  • For skill behaviour changes: a new or updated eval fixture is included in this PR
    (a regression test for the bug fixed / the behaviour added — see CONTRIBUTING.md)
  • Other: ran over existing skills

@justinmclean justinmclean self-assigned this Jun 30, 2026

@potiuk potiuk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cool :)

Add validate_modes_doc_consistency (SOFT, category modes-doc-consistency)
to the skill-and-tool-validator. The check compares docs/modes.md against
live skills/*/SKILL.md frontmatter across four dimensions:

1. Listed skill missing from disk — row exists in modes.md but
   skills/<slug>/ does not.
2. Mode mismatch — a skill's mode: frontmatter differs from the section
   it appears in (e.g. listed under Triage but mode: Mentoring).
3. Skill count mismatch — the integer in "Modes at a glance" Skill count
   column does not equal the actual row count in that mode's section.
4. Unlisted skill — a live skill has a valid mode: frontmatter value but
   is absent from the corresponding docs/modes.md section.

On the live tree the check surfaces two real gaps:
- reviewer-routing has mode: Triage but is absent from docs/modes.md
- good-first-issue-sweep has mode: Mentoring but is absent from docs/modes.md

Ships 19 tests in TestParseModesDocs + TestValidateModeDocConsistency;
all 283 validator tests pass.

Generated-by: Claude (Opus 4.7)
@potiuk potiuk force-pushed the modes-doc-consistency-check branch from 38a5ba4 to d13d6fc Compare June 30, 2026 23:45
@potiuk potiuk merged commit 7f7cb8c into apache:main Jun 30, 2026
30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants