feat(tools): deterministic vendor-neutrality score (magpie-site#17)#672
Merged
Conversation
Add `tools/vendor-neutrality-score`: a stdlib-only uv tool that scores Magpie's vendor neutrality straight from repository metadata, so the number is reproducible and cannot drift from the code. Answers apache/magpie-site#17 with a hard, falsifiable rule per capability contract (substrates excluded): - vendor-backed -> GREEN once >=2 distinct backend vendors implement it - agnostic -> GREEN by construction (one spec serves every backend) - single-org -> GREEN by exemption (bound to one org's data model) To make it deterministic, each of the 16 contract tools now declares `**Kind:**` (interface|implementation) and `**Vendor:**` in its README; `tools/ponymail` is additionally tagged `contract:mail-source` (it was already an inbound list-traffic source). The tool also assesses every skill: capability-pure / portable / vendor-coupled, plus declared `organization:` scope. docs/vendor-neutrality.md gains a "Vendor-neutrality score" section that describes the algorithm and embeds a generated block; a pytest keeps the block in sync with the tool output. Current score: 8/9 contracts vendor neutral (89%) — only mail-draft (Gmail-only outbound) is a gap. 59/63 skills carry no vendor lock-in; the 4 that do all reach mail-draft.
This was referenced Jul 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Answers the recurring "is Magpie really vendor neutral?" question from
apache/magpie-site#17 with a number that isn't a matter of opinion. Adds
tools/vendor-neutrality-score: a stdlib-only, offlineuvtool thatscores vendor neutrality straight from repository metadata, so the result
is reproducible from the source tree and can't drift from the code.
The rule
Neutrality is measured per capability contract (substrate tools —
Magpie's own machinery — are excluded). One rule per contract class:
it (one backend is a default, not neutrality; interface specs don't count)
Overall score = green contracts / total. Each skill is then classified
capability-pure / portable / vendor-coupled, with declared
organization:scope reported as an orthogonal dimension.To make it deterministic
**Kind:**(
interface|implementation) and**Vendor:**in its README.tools/ponymailis additionally taggedcontract:mail-source— it wasalready an inbound list-traffic source, just not tagged (capability-sync
tables updated to match).
docs/vendor-neutrality.mdgains a Vendor-neutrality score section(algorithm + a generated block); a pytest keeps that block in sync with
the tool output, so the published number can't go stale.
Result today
8/9 capability contracts vendor neutral (89%). The only gap is
contract:mail-draft(outbound mail is Gmail-only) — one adapter away.59/63 skills carry no vendor lock-in; the 4 that do all reach the same
mail-draftcontract.uv run --project tools/vendor-neutrality-score vendor-neutrality-score uv run --project tools/vendor-neutrality-score vendor-neutrality-score --json uv run --project tools/vendor-neutrality-score vendor-neutrality-score --markdown # regenerates the doc blockCloses apache/magpie-site#17 once the site tagline links here.