feat(labels): capability taxonomy + validator enforcement + sync check by potiuk · Pull Request #340 · apache/magpie

potiuk · 2026-05-27T10:35:56Z

Summary

Introduces a capability:* label dimension orthogonal to the
existing area:* labels, makes it enforceable across the
framework, and adds a sync check that keeps the docs and the
source aligned.

area:* answers what part of the framework does this touch?;
capability:* answers what kind of action does this perform?.
A single query on capability:triage finds every triage-flow
issue across area:pr-management + area:issue + area:security
without enumerating per-area.

What's in this PR

Taxonomy

New doc docs/labels-and-capabilities.md — canonical
reference: label-dimension definitions, 9 capability bucket
definitions, per-skill map for all 30 skills, per-tool map for
all 18 tools, the rule.
9 capability buckets: triage, review, fix, intake,
reconciliation (new — covers #337's
ASF-dashboard step), resolve, reassess, stats, setup.
Multi-capability examples: security-issue-fix =
fix+resolve (opens the PR that closes the tracker);
setup-isolated-setup-doctor = setup+reassess; cve-org =
resolve+intake; mail-source / ponymail = setup+intake;
skill-evals / spec-status-index = setup+stats.

Rule

AGENTS.md gains a new section that points at the canonical doc
and states the rule per artefact type: issues / PRs get
area:* + capability:*; new tools declare their capability in
the README first paragraph; new skills declare it in SKILL.md
frontmatter (single string or YAML list).

Backfill

30 SKILL.md files — capability: field added to every
existing skill's frontmatter via a one-shot script aligned with
the per-skill map in the doc. Dual-capability skills use the
YAML list form.
10 tool READMEs — **Capability:** line added in-place.
8 new tool README stubs — created for tools that previously
had no top-level README (cve-org, dev, github, gmail,
mail-source, ponymail, privacy-llm, vulnogram). Each
stub carries the capability line + a brief description that
points at existing internal docs.

Validator (renamed `skill-validator` → `skill-and-tool-validator`)

The validator grew beyond skills, so the directory, Python module,
CLI entry point (skill-validate → skill-and-tool-validate),
and every cross-reference (.asf.yaml,
.pre-commit-config.yaml, dependabot.yml, tests.yml,
CONTRIBUTING.md, write-skill, init_skill.py) were renamed.

Three new checks, all HARD:

Frontmatter capability — capability: is now required;
accepts single-string or YAML-list form; rejects values outside
the 9-bucket taxonomy.
Tool README + capability — every tools/<name>/ must have
a README.md with a **Capability:** capability:NAME line.
Multi-value form **Capability:** capability:NAME + capability:NAME
is supported.
Capability sync — compares the two tables in
docs/labels-and-capabilities.md against the live frontmatter +
tool README declarations, bidirectionally. Drift in either
direction is flagged. Italic-parenthetical future-state notes
(*(+ capability:X once #N lands)*) in the doc are stripped
before comparison.

Prek hook

The skill-and-tool-validate hook trigger now fires on changes
to tools/*/README.md and docs/labels-and-capabilities.md, not
just skill files, so the sync check catches drift at commit time.

Tests

12 new tests across TestValidateFrontmatter,
TestValidateTools, TestValidateCapabilitySync. Existing tests
updated to seed a minimal docs/labels-and-capabilities.md in the
test fixture (capability-sync is HARD and would otherwise fire
"missing doc" on every TestMain run). 218 tests green.

Scope notes

Adopter trackers are explicitly carved out — their label
schemes are per-project. The taxonomy applies to this
framework repository only.
Pre-existing soft warnings on 4 unrelated skills
(gh-list-no-limit, criteria-source) are not from this PR
and remain as-is.

Test plan

uv run --project tools/skill-and-tool-validator skill-and-tool-validate
clean (4 pre-existing soft warnings on unrelated skills).
pytest clean (218 tests).
Open the rendered doc on GitHub and spot-check table
alignment and link rendering.
Sanity-check the prek hook firing on a sample doc change
in the next branch.

Branch protection on `main` cannot use patterns for required status check names — GitHub's classic branch-protection and rulesets APIs both require exact-match `context` strings, so each entry in the `pytest` matrix in `tests.yml` previously needed its own line in `.asf.yaml`. That coupling caused two known issues: - PR #340 renames `pytest (skill-validator)` → `pytest (skill-and-tool-validator)` and cannot merge, because branch protection on `main` still requires the old context name (which no longer runs in CI after the rename). - The `pytest (agent-isolation)` matrix entry added in #339 was never added to `.asf.yaml` and is silently not gated. Add a `tests-ok` umbrella job in `tests.yml` that `needs:` the `pytest` matrix and fails unless every matrix entry succeeded. Replace all `pytest (...)` entries in `.asf.yaml` with the single `tests-ok` context. Adding, renaming, or removing matrix entries no longer touches `.asf.yaml`. Follow-up for #340: drop the `.asf.yaml` hunk (the line is gone on `main`) and rebase. Generated-by: Claude Code (Opus 4.7)

Introduce a canonical capability taxonomy for the framework, enforce it in the validator, and add a sync-check that keeps the docs and the live source aligned. == Taxonomy == Nine `capability:*` buckets orthogonal to the existing `area:*` labels: triage, review, fix, intake, reconciliation (new — covers and tools may carry more than one capability when they genuinely span lifecycle phases — `security-issue-fix` is fix+resolve, `setup-isolated-setup-doctor` is setup+reassess, `cve-org` is resolve+intake, etc. docs/labels-and-capabilities.md is the canonical reference: label- dimension definitions, capability bucket definitions, per-skill map for all 30 skills, per-tool map for all 18 tools. == Rule == AGENTS.md states the rule: every issue, PR, new tool, new skill, and (where applicable) new doc declares its capabilities. - Issues / PRs: `area:*` + every applicable `capability:*` label. - New tools: `**Capability:** capability:NAME` (or `capability:NAME + capability:NAME` for multi-value) in the README first paragraph. - New skills: `capability:` in SKILL.md frontmatter — single string or YAML list form. `setup-override-upstream` now picks both labels before opening a framework PR. `write-skill` requires the new frontmatter field on every new-skill scaffold. == Backfill == - All 30 existing skills got `capability:` added to their frontmatter via a one-shot script aligned with the per-skill map. - 10 existing tool READMEs got a `**Capability:**` line. - 8 tools without a README got a minimal stub README declaring capability + pointing at existing internal docs. == Validator (renamed skill-validator -> skill-and-tool-validator) == The validator outgrew its name. Renamed the directory, the Python module, the CLI entry point (`skill-validate` -> `skill-and-tool-validate`), and updated every cross-reference (.asf.yaml, .pre-commit-config.yaml, dependabot.yml, tests.yml, CONTRIBUTING.md, write-skill, init_skill.py). New checks: - `capability` is now in REQUIRED_FRONTMATTER_KEYS. Validates both single-string and YAML-list forms; rejects values outside the 9-bucket taxonomy. - `validate_tools()` — every `tools/<name>/` must have a README declaring its capabilities. Both "missing README" and "missing capability line" are HARD violations. - `validate_capability_sync()` — compares the two tables in docs/labels-and-capabilities.md against the live frontmatter + tool README declarations, bidirectionally. Drift in either direction is a HARD violation. Italic-parenthetical future-state notes (`*(+ capability:X once #N lands)*`) are stripped before comparison so the doc can flag planned capabilities without tripping the check. Prek hook trigger expanded so the sync check fires on `tools/*/README.md` and `docs/labels-and-capabilities.md` changes, not just skill files. == Tests == 12 new tests across 3 classes — single + list + missing + invalid + list-with-invalid for the frontmatter check; valid + missing- readme + missing-cap + invalid + multi-value + regex regression guard for the tool check; aligned + skill-doc-no-live + live-skill- no-doc + skill-mismatch + tool-doc-no-live + live-tool-no-doc + italic-parens for the sync check. All 218 tests green. Generated-by: Claude Code (Opus 4.7)

The capability-sync check from apache#340 requires every tool with a `**Capability:**` declaration to carry a row in `docs/labels-and-capabilities.md`, and every tool README to declare its capability. `tools/spec-validator/README.md` predates the rule; add the line (matches sibling meta tools — `skill-and-tool-validator`, `spec-loop` — at `capability:setup`) and the corresponding table row. Also fix a stale ref in the README: `tools/skill-validator/` → `tools/skill-and-tool-validator/` (renamed in apache#340). Generated-by: Claude Code (Opus 4.7)

* feat(meta): add spec-validator tool — validate spec frontmatter and body sections Adds tools/spec-validator/, a stdlib-only uv tool analogous to tools/skill-validator/ that validates every .md file carrying a YAML frontmatter block in tools/spec-loop/specs/: 1. Required frontmatter keys (title, status, kind, mode, source, acceptance) 2. Valid status / kind / mode values 3. Non-empty acceptance list 4. Required body sections (What it does, Where it lives, Behaviour & contract, Out of scope, Acceptance criteria, Validation) 5. Validation section contains at least one fenced code block Files without frontmatter (README.md, overview.md) are silently skipped. 56 tests pass; 11 live specs from the control branch validate clean. Note: all 7 IMPLEMENTATION_PLAN.md items were found to be merged or in-flight before this iteration; this item was derived from the Known gap in specs/meta-and-quality-tooling.md ("a spec validator analogous to the skill validator"). A plan/update beat should reconcile. Generated-by: Claude (Opus 4.7) * chore: declare spec-validator capability + sync capabilities doc The capability-sync check from #340 requires every tool with a `**Capability:**` declaration to carry a row in `docs/labels-and-capabilities.md`, and every tool README to declare its capability. `tools/spec-validator/README.md` predates the rule; add the line (matches sibling meta tools — `skill-and-tool-validator`, `spec-loop` — at `capability:setup`) and the corresponding table row. Also fix a stale ref in the README: `tools/skill-validator/` → `tools/skill-and-tool-validator/` (renamed in #340). Generated-by: Claude Code (Opus 4.7) * chore: clear codeql findings on test file (unused imports + var) CodeQL flagged two unused names in `tests/test_spec_validator.py`: - unused imports `REQUIRED_FRONTMATTER_KEYS` and `validate_file` - unused local variable `text` in `test_missing_code_block` (left over from an earlier draft; the test already uses `spec_no_code` for the actual assertion) Remove all three. Ruff clean; 57 tests still pass. Generated-by: Claude Code (Opus 4.7) --------- Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

* Update apache-steward snapshot to 5c211a4 Bumps the local apache-steward snapshot from 339d3eb to 5c211a4 (22 upstream commits). The only committed change in this PR is a 1-line frontmatter addition (capability: capability:setup) to .github/skills/setup-steward/SKILL.md, propagated from the new framework version via /setup-steward upgrade. Everything else lives in the gitignored .apache-steward/ snapshot. Highlights from upstream (apache/airflow-steward): - pr-management-triage: session-history gist persistence Step 6b (apache/magpie#343), four classifier heuristic fixes (apache/magpie#344), fetch-all-upfront pattern (apache/magpie#346) - security-issue-triage: fetch-all-upfront analogue (apache/magpie#347) - Framework labels + capability taxonomy (apache/magpie#340) — the source of the frontmatter line in this PR - New skill pairing-self-review and tool spec-status-index - claude-code pin 2.1.141 -> 2.1.150 /setup-steward upgrade ran cleanly locally: snapshot refreshed, symlinks resolve, post-checkout hook in sync, sandbox-add-project-root reconciled across 3 worktrees. .apache-steward.local.lock updated to fetched_commit 5c211a4. All .apache-steward-overrides/ files unchanged. * Gitignore .apache-steward.session-state.json Adds the per-machine session-state file to .gitignore. The file is written by steward skills that maintain adopter-local persistence anchors — currently pr-management-triage Step 6b's session-history gist URL (apache/magpie#343), but the structure is deliberately shared so other skills can add their own keys later. The file is per-user, per-machine state; it should never be committed even when a contributor stages everything with `git add -A`.

* Update apache-steward snapshot to 5c211a4 Bumps the local apache-steward snapshot from 339d3eb to 5c211a4 (22 upstream commits). The only committed change in this PR is a 1-line frontmatter addition (capability: capability:setup) to .github/skills/setup-steward/SKILL.md, propagated from the new framework version via /setup-steward upgrade. Everything else lives in the gitignored .apache-steward/ snapshot. Highlights from upstream (apache/airflow-steward): - pr-management-triage: session-history gist persistence Step 6b (apache/magpie#343), four classifier heuristic fixes (apache/magpie#344), fetch-all-upfront pattern (apache/magpie#346) - security-issue-triage: fetch-all-upfront analogue (apache/magpie#347) - Framework labels + capability taxonomy (apache/magpie#340) — the source of the frontmatter line in this PR - New skill pairing-self-review and tool spec-status-index - claude-code pin 2.1.141 -> 2.1.150 /setup-steward upgrade ran cleanly locally: snapshot refreshed, symlinks resolve, post-checkout hook in sync, sandbox-add-project-root reconciled across 3 worktrees. .apache-steward.local.lock updated to fetched_commit 5c211a4. All .apache-steward-overrides/ files unchanged. * Gitignore .apache-steward.session-state.json Adds the per-machine session-state file to .gitignore. The file is written by steward skills that maintain adopter-local persistence anchors — currently pr-management-triage Step 6b's session-history gist URL (apache/magpie#343), but the structure is deliberately shared so other skills can add their own keys later. The file is per-user, per-machine state; it should never be committed even when a contributor stages everything with `git add -A`. (cherry picked from commit c521078) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>

potiuk force-pushed the feat-labels-and-capabilities-taxonomy branch 2 times, most recently from b8bb856 to a363653 Compare May 27, 2026 18:00

potiuk mentioned this pull request May 27, 2026

chore(ci): require pytest via tests-ok umbrella job #341

Merged

4 tasks

potiuk force-pushed the feat-labels-and-capabilities-taxonomy branch from a363653 to 6ea6554 Compare May 27, 2026 18:55

potiuk merged commit 24ddd89 into apache:main May 27, 2026
17 checks passed

potiuk mentioned this pull request May 27, 2026

Update apache-steward snapshot to 5c211a4 apache/airflow#67624

Merged

1 task

potiuk mentioned this pull request May 27, 2026

feat(meta): add spec-validator tool #336

Merged

15 tasks

potiuk mentioned this pull request May 28, 2026

fix: update stale skill-validator references to skill-and-tool-validator #352

Merged

github-actions Bot mentioned this pull request May 28, 2026

[v3-2-test] Update apache-steward snapshot to 5c211a4 (#67624) apache/airflow#67665

Merged

potiuk mentioned this pull request May 30, 2026

feat(security): forwarder-relay + mail-archive sub-tools (PR3/5) #387

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(labels): capability taxonomy + validator enforcement + sync check#340

feat(labels): capability taxonomy + validator enforcement + sync check#340
potiuk merged 1 commit into
apache:mainfrom
potiuk:feat-labels-and-capabilities-taxonomy

potiuk commented May 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

potiuk commented May 27, 2026

Summary

What's in this PR

Taxonomy

Rule

Backfill

Validator (renamed skill-validator → skill-and-tool-validator)

Prek hook

Tests

Scope notes

Test plan

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Validator (renamed `skill-validator` → `skill-and-tool-validator`)