[DO-NOT-MERGE] trial branch: all skill PRs merged, for install and testing - #108
Draft
MajorLift wants to merge 241 commits into
Draft
[DO-NOT-MERGE] trial branch: all skill PRs merged, for install and testing#108MajorLift wants to merge 241 commits into
MajorLift wants to merge 241 commits into
Conversation
sentry-mcp-queries: document data-fidelity loss on older releases for longer-range (30D+) queries (sample-rate drift, extrapolation hiding thin samples, retention downsampling) and percentile (p75+) sample-size/quality filtering; add table rows + pitfalls for stored-span-count and superseded-patch releases. performance-attribution: new skill for attributing release-over-release p75/p95 movements to code changes via black-box diff analysis, with an extension repo file (Trace Explorer queries, key transactions, highest-sample-patch version selection, 90d-vs-30d empirics, hot-path files, core-package changelog analysis, worked v13.11->v13.15 catalogue). analytics-instrumentation: cross-link volume-estimation caveats to the new fidelity guidance. CHANGELOG updated.
…n-out, latent trace? param, trace-arg PR-review scan, per-name budget
…cidental instrumentation)
…-types, decompose-large-files
…ary-identification is step one of any migration
…d exception (bivariant callback `any`)
…und unit conversion, swap in the stronger derive-types example, slim CHANGELOG - migration-context-cost: line count is a factor not a non-factor; fan-in is a reading cost (not a change/review surface), fan-out is the change surface; drop the wrong "upstream types land first" and off-topic barrel bullet - decompose-large-files: the point is converting to TS in small self-contained units, not extraction - derive-types: replace the NetworkState restatement with the reinvented-messenger + hand-copied-return example (derive via `ReturnType<Action['handler']>`) - CHANGELOG: list the domain, not each skill
…action is optional
Two-phase retention review for JavaScript/TypeScript. Phase 1 is a static read of a diff: enumerate the retention primitives the change introduces — listeners, timers, pending-request registries, subscriptions, module singletons, growing collections — and pair every acquire with its release site. A primitive with a teardown is safe; one without is the finding. Phase 2 escalates to DevTools/CDP heap snapshots only for a primitive the read cannot pair. Leading with the read rather than the instrument settles most leak claims without ever taking a snapshot.
…oat-policy-diligence Two composing skills for "is this dependency change safe to take". `supply-chain-audit` is the breadth pass: Socket findings, `yarn npm audit` advisories, lockfile and manifest diffs, and the fronts no upstream scanner sees because they are things the repo does to its dependencies afterwards — yarn patches that modify dependency source at install, `resolutions` that force or stub versions, `npmAuditIgnoreAdvisories` suppression lists, CI actions riding mutable tags, and yarn plugins that execute at install. `lavamoat-policy-diligence` is the depth pass it delegates capability containment to. Because a LavaMoat policy is generated from a real run, every grant has a call site by construction — so "each addition is justified" is a tautology, not a finding. It instead reads each grant's use at the installed version to find its gate, and sorts into removable / removable-at-a-cost / load-bearing. Neither renders an accept/reject verdict; disposition belongs to the people who own the dependency. Adds a CODEOWNERS entry for the new domain, defaulted to the platform teams.
Replaces #56, whose skill.md shipped frontmatter with no body at all — the installer's bodyAfterFrontmatter() returned empty, so an agent loading it got a description and six references it had no instruction to read. pr-validate: for a PR's specific falsifiable claim, name the observation that would prove the claim false, gather it, and publish it into the PR body. Drives the AEP harness (visual_validation, perf_validation) as the primary engine, backed by a catalog of complementary lanes, a trustworthiness gate that rejects vacuous passes, and a publishing flow with an audience-reachability rule for re-hosted artifacts. falsifying-test: the strongest single proof that a fix targets the reported bug — a test that fails on the base commit and passes on the branch, with both runs shown. Its own falsifier is a base-commit failure for the wrong reason (import error, missing fixture, unrelated red), which looks identical in an exit code and proves nothing. pr-validate calls it as the engine behind lane B3. hooks/pr-evidence-gate.py enforces the trustworthiness gate at emit time, blocking an outward-facing write whose body carries an unbacked verdict, an untracked deferral, a CI restatement, a bare or truncated identifier, a mutable ref, a dump-as-resolver, a link-only or data-only exhibit, or a step waiver. It polices `gh api` body writes as well as the porcelain, since a PATCH to a comment is the same publish with a different spelling.
CHANGELOG.md tracks consumer-facing changes to the `@metamask/skills` package, per CONTRIBUTING's "CLI / tooling changes" section. No merged skill-only PR adds an entry (#80, #78, #70, #62, #61 all touch zero changelog lines). Carrying one here bought nothing and was the sole source of this branch's conflict with `main`, since every skill PR edits the same `[Unreleased]` block. Restoring the file to its merge-base state makes the branch conflict-free without a merge commit.
CHANGELOG.md tracks consumer-facing changes to the `@metamask/skills` package, per CONTRIBUTING's "CLI / tooling changes" section. No merged skill-only PR adds an entry (#80, #78, #70, #62, #61 all touch zero changelog lines). It was also the sole source of this branch's conflict with `main`, since every skill PR edits the same `[Unreleased]` block.
CHANGELOG.md tracks consumer-facing changes to the `@metamask/skills` package, per CONTRIBUTING's "CLI / tooling changes" section. No merged skill-only PR adds an entry (#80, #78, #70, #62, #61 all touch zero changelog lines). It was also the sole source of this branch's conflict with `main`, since every skill PR edits the same `[Unreleased]` block.
The selector and effect anti-pattern definitions existed in two places: these knowledge files, and the `performance` skill's own mm-* references already on main. Same patterns, same worked examples, two homes that would drift. knowledge/selector-anti-patterns.md and knowledge/effect-anti-patterns.md are now the canonical, platform-agnostic taxonomy — the union of both sides. The selector file absorbs mutation-in-result and over-broad-input from mm-selector-memoization; the effect file absorbs the dependency-side patterns from mm-hook-dependency-arrays and the lifecycle-side patterns (derived state, effect chains, uncancelled async). mm-selector-memoization.md keeps everything only it can say — the codebase's own selector creators, the verified instance table with file:line, the fix recipes, the scoped greps, the don't-over-correct caveats — and maps each generic pattern onto this codebase instead of redefining it. mm-hook-dependency-arrays.md keeps its richer JSON.stringify treatment and gains a scope note. Citations are by NAME, not by relative link. `install` copies domain knowledge/ and a skill's references/ as siblings under the installed skill directory, so `../../../knowledge/x.md` resolves in the repo and breaks once installed, and `../knowledge/x.md` does the reverse. Section anchors are dropped for the same reason — they broke the moment the taxonomy was renumbered. Also drops the CHANGELOG entry: that file tracks the @metamask/skills CLI package, no merged skill-only PR adds one, and it was this branch's sole conflict with main.
Folds in the react-render-proof skill (was #82) and the mobile reference-library additions (was #49). All three were the same effort seen from different ends — moving performance work earlier in the loop — and they share a substrate, so reviewing them apart meant reviewing the substrate three times. The loop this domain now covers: - catch it at review — effect/selector anti-pattern review skills, driven by the knowledge taxonomy - prove it moved — react-render-proof, with a delivery gate so an arm whose treatment never reached the bundle cannot report as a null - measure it honestly — data-analysis, benchmark hygiene, web-vitals framing - know the codebase — the mm-* reference library and its audit playbook Also neutralizes five references to private planning tickets, which do not belong on a public repository — they named internal epic and audit-ticket numbers. The surrounding guidance is unchanged; only the identifiers are gone.
These name MetaMask-org planning epics and audit tickets. The audience for this repo is the MetaMask org, for whom those identifiers are load-bearing context — they are where the guidance came from and where the follow-up lives. The scrub line is personal references, not org-internal ones.
Over-scrubbed. The audience is the MetaMask org, and the ticket is the evidence for the claim the item makes — that the LaunchDarkly provisioning blocker covers only the prod-flag half of that lane. Without it the example is an assertion. The scrub line is personal references, not org-internal ones.
The skill covers runtime retention behaviour, not code authoring, and `coding` reads as language- and style-level guidance. Registers `/domains/stability/` in CODEOWNERS alongside the other platform-owned domains.
…main Was a separate PR against `domains/testing`. It belongs here: its subject is whether a hand-written type agrees with its authoritative source, which is the question `derive-types` answers from the authoring side, and it shares this domain's premise that a green `tsc` is not evidence the types are correct. Directory name and frontmatter `name` already agree; only the domain moved.
Renamed on the security-domain branch; installs as `mms-lavamoat-policy`.
A diligence comment renders no verdict and deliberately does not use the Validation Run envelope. That exemption meant it was checked by nothing: this gate only knew the Validation Run shape, and `pr-evidence-gate.py` by design does not trip on a body claiming no verdict. So every rule the diligence skills state about their own output — including "runtime claims need a runtime artifact" — had no execution path. It showed. A lavamoat comment shipped with no marker pair, an `npm pack` specifier set no reader could fetch, and two bare integers traceable to nothing. `--diligence` swaps the four envelope checks for that contract's own — its marker pair, its header, permalinks pinned to a tag or SHA rather than a branch head, and a runtime claim check asking for the thing a `/blob/` link cannot witness. 3, 8 and 9 report SKIP with the reason rather than passing silently, since a check that cannot fail should not read as a check that passed. Everything downstream of the envelope is shared, because those defects are shared. Run against the comment that prompted this, it fails 1, 5 and 13 and passes the rest.
This repository is public. Naming a private repository here discloses its existence,
its owner and roughly its contents to every reader — and a prohibition naming it
("do not re-host to X, it is private") discloses exactly as much as a recommendation
would. Four such references were doing that, and the guidance survives without them:
the rule is audience-reachability, which is stated directly rather than by example.
Two memory-file citations offered as "source of truth" pointed into a private repo, so
a reader was told to follow a rule whose justification they cannot open. The reasoning
is inlined; the pointer is gone.
The publish-surface snippet hardcoded a GitHub username, which decided the destination
for whoever ran it. Now derived from `gh api user --jq .login`, and the surrounding
prose is second-person rather than first — a shared skill has no "my PRs".
`/attest` is no longer linked to a personal repository. That leaves it named but not
resolvable, which is honest and is the smaller problem; the workflow depending on a
command nobody else has is tracked separately.
The re-hosting section named a personal S3 bucket, its region, its prefix layout, and stated that anonymous `GetObject` is enabled under `public/*`. That is a live unauthenticated endpoint advertised, with its structure, to every reader of a public repository — a larger disclosure than the repository names removed alongside it, and one that reads as configuration rather than as a secret, which is why it survived two passes over this file. Now `EVIDENCE_BUCKET` and `EVIDENCE_REGION` from the environment. The requirements the bucket must satisfy — anonymous GetObject under `public/*`, listing disabled — are stated, because those are the load-bearing part; the name never was. The G5 lane likewise named a private test fork, which carried both the org and a personal handle. Now "your own test fork".
…setup Removing the hardcoded username left `gh pr view --jq --arg me "$ME"`, which is not a thing gh supports — its built-in filter takes no --arg and the command dies with "accepts at most 1 arg(s)". Piped to real jq instead, and checked against both branches of the logic: a PR authored by someone else resolves to "skip", one authored by the caller to "body". Replacing the named bucket with `EVIDENCE_BUCKET` removed a working default and put nothing in its place, so the section told you to configure a bucket without saying what "conforming" meant. The policy is now stated: anonymous `s3:GetObject` under `public/*`, public-access blocks off for that bucket, `s3:ListBucket` to nobody. With a note that an org-owned bucket beats a personal one, since artifact links outlive their publisher.
A results section reached a public PR whose entire content was hand-typed to look like
terminal output. Three independent things had to hold for that, and all three did.
The gate is model-invoked, so it can be skipped: the publish and the gate ran as two
statements rather than one chain, and the verdict was read after the write.
The hook that fires on the publish call carried a SECOND, narrower copy of the rules —
keyed on verdict tokens — so a comment rendering no verdict satisfied neither copy.
Two rule sets means the weaker one governs whatever falls between them. The hook now
delegates to `attest-gate.sh`: one rule set, invoked by construction rather than by
choice, in the mode the body's markers imply. It fails CLOSED once it has identified a
body it is about to publish — an enforcement point that waves things through when it
cannot find its rules is not one.
And check 5 in `--diligence` had been rewritten as a phrase denylist ("npm pack",
"complete specifier set"), which is precisely the regression its own comment records as
having shipped four times: every property of plaintext is forgeable by whatever emits
the plaintext. It is a medium test again — if the artifact shows a command or a run
result, it owes the reader something fetchable. `/blob/` links are excluded, because a
permalink to a `.json` file satisfied a naive extension test and was the specific reason
the hand-typed section passed.
Comment-update URLs carry the comment id, not the issue's, so check 12 was asking
whether pull #5177261620 was open. Resolved through the API instead.
Four-arm verified: blocks the exact command and body that shipped; ignores `ls`; ignores
a `gh` read with no body write; refuses when the gate is unreachable.
`mms-evidence` named `/attest` as the gate its output passes and shipped no such thing. Anyone installing the skill got phase 0 as a script and a reference to a command only its author had, which is half a publish path for everyone else. Phase 0 is `attest-gate.sh`, already here. Phase 1 is three briefs sent to fresh instances — frame, coverage, and how it reads to a stranger — written out in `references/dispatched-passes.md` so dispatch does not depend on commands that live elsewhere. The check table in `references/phase-0-checks.md` is generated from the gate rather than retyped, so it cannot drift from what runs. Two things the port makes explicit that the original left to discipline. The gate must be the same shell chain as the publish, because running both and reading the verdict after the write is how a blocked artifact reached a public PR. And softening a check to fit the case in hand is called out as an anti-pattern: if the new version could be satisfied by better prose alone, it is no longer the check.
Three copies of this hook were on one machine and the oldest was the one wired into settings — no `gh api` matcher, no delegation to `attest-gate.sh`. Every publish through that path went ungated, and nothing noticed, because a gate that blocks nothing looks exactly like a gate with nothing to block. `gate-controls.sh` is the thing that would have noticed. Six arms: three publish routes that must block, three inputs that must pass. It copies the hook somewhere with no sibling `scripts/` so `_find_gate()` resolves the way it does in production rather than the way it does in a checkout — the difference matters, and testing the checkout copy is how the deployed one stayed broken. Both halves earned their place immediately. The negative arm caught that delegating to attest-gate ran it over EVERY published body, so an ordinary reply was judged as a failed validation run; wiring that would have blocked every normal comment. The gate now applies only to bodies carrying an artifact marker or a verdict line. And the positive arm caught itself: the enrichment probe was a single sentence, which that rule correctly ignores, so the arm had been passing because attest-gate blocked the body for an unrelated reason. A probe that fires for the wrong reason reports a working rule. The `enrichment` class is ported forward from the older copy, with the constant it depends on — it existed in the deployed version and in neither newer one.
The six arms prove the script blocks what it should. They say nothing about whether anything calls it, and those are different questions — a hook that is unwired, or wired to a path that no longer exists, is indistinguishable from a hook with nothing to block. One session ran start to finish with every PreToolUse hook inert: 306 certification markers written, none enforcing anything, and the ritual read as compliance. The check enumerates config roots rather than trusting `$HOME`. Its first version did trust it, found one settings file, reported it as "the" wiring and never looked at the second — because `$HOME` here points at a per-account directory rather than the login home. That is the same defect one level up, caught only because two configs were known to exist and one was missing from the output. It stops short of claiming liveness, and says so: a settings file naming an existing file is not proof the running session loaded it. Only a command the gate must block, issued in a session and observed to be blocked, shows that.
This is the bypass. The hook reads the command as text, so `--body-file $DIR/comment.md` resolves to nothing — and the code called that "can't read it -> nothing to scan" and allowed the write. Every publish in one long session used a shell variable for the path, so every one of them went ungated, including a comment whose entire results section was hand-typed to look like terminal output. That same body, passed by literal path, is blocked. An unreadable body is not an absent risk. By that point the command is already identified as an outward-facing write; not knowing what it carries is the reason to stop. Also rejects a body argument carrying `$` or a backtick — `--body "$(cat f)"` extracts the literal characters, scans clean, and publishes whatever the shell substitutes afterwards. The check reads the command ARGUMENT, not the body. A first version scanned body text for shell metacharacters and blocked every evidence comment ever written, because markdown inline code is backticks and these artifacts are full of them. Caught by the negative arm, which is the half of a control matrix that earns its place on days like this. Three arms added, verified end to end: the command that slipped through minutes earlier is now stopped by the deployed hook.
This branch adds `hooks/pr-evidence-gate.py` and `hooks/gate-controls.sh`, and on main neither would have survived: `hooks` is absent from `BUNDLE_DIRS`, so the linter rejects the directory outright, and absent from the bundle loop in `tools/install`, so nothing would copy it even past lint. The gate would have merged into the repository and installed to nobody. `hooks` now appears in both. The third change is the one worth keeping. The two lists are a single contract written twice, in JavaScript and in Bash, and the schema's own comment asked humans to keep them in sync by hand. `tools/check-bundle-contract.sh` asserts it instead. It builds a fixture skill carrying one file in every declared bundle directory, runs a dry-run install from a copy of the installer placed inside that tree, and checks each directory is reported as copied. It watches behaviour rather than reading the loop, because a regex over the source is the same class of evidence as the defect: two declarations agreeing with each other and neither of them being what runs. Controlled both ways. It passes as committed. Removing `hooks` from the loop makes it name `hooks` and exit 1; removing three makes it name three. Also relocates `probes/metametrics-context.test.tsx`, which lint rejected as an unexpected directory, to `assets/` where a copy-out template belongs, and points the C4 lane at it. It had been shipping with nothing in the skill naming it, which is the same defect one layer over: present, and unreachable.
A check nobody invokes is a check that fires once, on the day it was written. This one guards a contract that drifts silently: a bundle directory declared in `BUNDLE_DIRS` and missing from the install loop reaches no consuming repo, and nothing errors, because the skill still installs, still loads, and still answers without whatever that directory held. It runs in the lint job, after `yarn lint` and before the changelog steps. It leaves no files behind, so the job's clean-working-directory gate is unaffected.
tools/install copies source over the installed tree with `rm -rf` followed by `cp -R`, so an edit made in the installed copy is destroyed on the next sync with no warning and no record. Comparing source against the install cannot tell the two cases apart, because both make them differ: the install was edited locally, or the source moved ahead. A guard that refuses on any difference breaks every routine update, which is a good way to get itself disabled. So each install now records a content hash of what it wrote, and a later run compares the install against that record. Matching means nothing local happened, and the copy proceeds no matter how far source has moved. Bundles with no record predate the manifest and fall back to comparing against source. That is conservative — it refuses a legitimate update once, until a manifest exists — on the grounds that a one-time re-run beats a silent delete. Also adds hooks/ to BUNDLE_DIRS and to the directories tools/install copies. It was in neither, so a skill shipping a hook had it installed by hand and tracked nowhere. That is how the evidence skill's wired PreToolUse gate — the one control that fires on every publish — came to have no source of record. The existing lint-skill-entry test asserts these two lists agree, and caught the omission when only one was updated. Controls, against a temp target and temp HOME: clean install exit 0 re-install, nothing touched exit 0 source moves ahead exit 0, change propagates local edit to the install exit 1, REFUSED, sha unchanged same with --force exit 0, edit discarded `node --test test/*.test.mjs` 61/61.
The ARTIFACT and OBS_ARTIFACT classes matched `https?://\S+` and a bare filename pattern, so any URL-shaped string discharged them. The same generator writes the claim and the string that satisfies the check, which makes presence carry no information about whether the evidence exists. Measured against the unpatched hook, with the claim and the artifact in the same block (proximity matters — the checks are unit-scoped): https://example.invalid/nope/capture.png ALLOWED `src/totally/made-up.test.ts:42` ALLOWED `evidence/never-captured.png` ALLOWED An artifact now counts only if the author could not have authored its bytes: a namespace where CI, the upload endpoint or an observability backend writes them, or a local path that is actually on disk. github.com is author-writable in general, so only /actions/runs/, /user-attachments/, /blob/, /commit/ and /pull/ under it qualify. Controls, before -> after: fabricated (want blocked) 1/3 -> 3/3 genuine (want allowed) 2/2 -> 2/2 Replaying real published bodies found the interesting case: evidence hosted in an author-controlled S3 bucket is now refused, correctly — fetching it unauthenticated proves it is fetchable, not that the author did not write it. Rather than decide that silently, EVIDENCE_GATE_ARTIFACT_HOSTS registers such hosts explicitly. Doing so is a visible downgrade from independent to merely fetchable, which is the point: it should be a choice someone made, not a property of the regex. strict default -> those bodies block bucket registered -> those bodies pass, fabrications still block The rule is presence stays necessary and stops being sufficient. Note this is still not resolution: nothing yet fetches the URL and checks for a 200. An allowlisted-but-nonexistent /actions/runs/99999999999 link is refused by the separate CI-restatement rule rather than by this one.
The hook already ran attest-gate.sh in-process on the exact bytes about to be
published, which is a stronger binding than recording a verdict against a
hash — there is no window in which the artifact can be edited after the gate
passes. Two defects meant it often did not run, or ran the wrong gate.
1. _is_evidence_artifact matched `^\*\*Verdict:\*\*`, so where the bold
stopped decided whether thirteen checks ran:
**Verdict:** proven -> all 13 checks
**Verdict: proven** -> none
The same sentence, rendered identically, one of them silently unenforced.
Now matches a line-leading bolded Verdict however the emphasis falls, still
anchored so prose mentioning the word does not drag an ordinary reply in.
2. _find_gate ranked $ATTEST_GATE last, behind three default paths. An
override that loses to a default is not an override: a control run pointing
it at a stand-in silently exercised the installed gate and reported on that
instead. This is the failure the skill's own non-negotiable 9 describes —
an instrument reporting the instruction it was given rather than the effect
it had — and it was in the resolver for the gate itself.
Verified by substituting a stand-in gate whose exit code is controlled, with
the hook copied to a directory with no sibling scripts/ (as gate-controls.sh
does, for exactly this reason):
before after
stand-in exits 0 ALLOWED ALLOWED
stand-in exits 1 ALLOWED BLOCKED
gate path nonexistent ALLOWED BLOCKED
gate-controls.sh: all eleven arms behave, including the three negative arms
that catch a gate which has started blocking everything.
…ntmatter mean something Three controls that were present and inert. 1. evidence-run.yml's determinism check emitted ::warning:: and carried continue-on-error, so the step whose own message reads "do not publish these numbers" went green and the numbers published. Now ::error:: plus exit 1, with continue-on-error removed from that step only. The two runner steps keep it, because there the exit code is the verdict and a finding is not a failure. Here a difference means the instrument did not return the same answer twice, so neither answer is publishable. Verified by extracting the step and running it: identical arms exit 0 — including the label/log/logs/env fields it deletes by design, which is the false positive that taught operators to publish through it — and differing arms exit 1. 2. selector-recompute.sh returned 0 for every outcome including "VALUE UNSTABLE" and "probe-failed", so a caller gating on the exit code saw green on a run whose own artifact says the number is not meaningful. The only thing between that and publication was attest-gate happening to grep the verdict string out of the prose. Now 4 for VALUE UNSTABLE and 5 for probe-failed, with 0 kept for both real measurements — a selector that recomputes is a result, not an error. Codes documented in the header, and the --help range extended so it shows them. 3. Three of fifty skill.md frontmatters were invalid YAML: `description` was a plain scalar containing ": ", which YAML reads as a nested mapping. Any tool parsing source frontmatter fails on them; tools/install masked it by folding the value to a block scalar on the way out, so the installed copy parsed and the source did not. Folds evidence and attest to block scalars at rest. The performance skill has the same defect but lives on main, so it is fixed in the tools/install branch rather than here. Round-trip checked: the installed description is byte-identical to the source description after folding. `node --test test/*.test.mjs` 61/61.
`description` was a plain scalar containing ": ", which YAML reads as a nested mapping, so the frontmatter did not parse. Any tool reading source frontmatter with a YAML parser fails on it; tools/install masked the defect by folding the value to a block scalar on the way out, so the installed copy parsed and the source did not. Folded to a block scalar at rest. The description text is unchanged.
…aring a schema
The three .graph.json specs name an `executor` and nothing parses them. A run
could skip a node and still describe itself as having followed the graph,
because nothing held the description to the file — which is the failure the
whole package is about, one level up.
What this can and cannot do is worth being exact about, since overstating it
would reproduce that failure. The check, run and expect fields are natural
language; no interpreter evaluates them and this one does not pretend to. A
node's verdict is supplied by whoever ran it.
What is mechanised is the part that was actually being skipped:
- every node must carry a verdict
- a pass must name its evidence, because a pass with an empty evidence field
is a claim about a run, which is refused everywhere else in this package
- `absent` is a first-class outcome, reported rather than silently dropped
- a ledger entry naming no node in the graph is an error, not ignored
- a graph with an unanswered node cannot report complete
Structural validation refuses a graph missing a baseline or a prediction,
those being the shape of every vacuous run: a treatment applied to nothing, or
a result compared to no stated expectation.
graph-run.mjs <graph.json> validate structure
graph-run.mjs <graph.json> --scaffold emit a blank ledger
graph-run.mjs <graph.json> --ledger <l.json> check a filled-in ledger
Controls: all three graphs validate; blank ledger exit 1; fully answered exit
0; one node silently removed exit 1; pass with empty evidence exit 1; unknown
ledger entry exit 1.
… a claim Broadening the artifact trigger made it fire on text ABOUT the trigger. A pull request quoting `**Verdict:** proven` to show what the gate matches was classified as a validation run and asked for the whole envelope, so documenting the rule became a violation of it. Fenced blocks are stripped before the trigger is tested. real verdict in prose -> artifact real verdict, bold-wrapped -> artifact verdict ONLY inside a fence -> not an artifact fenced example alongside a real one -> artifact ordinary reply -> not an artifact explicit VALIDATION_RUN marker -> artifact gate-controls.sh 11/11; chaining controls still verified; fabricated-artifact controls still 3/3 blocked. Surfaced by the trigger now working: a real published body is blocked by the CI-restatement rule, which had never run against it because the old trigger did not match its verdict line. That rule and this package's own direction disagree, and the conflict is left for the owner rather than resolved here — see the pull request description.
The proposal was to cut the prose: every principle with a hook class becomes one line pointing at the class, leaving only irreducible judgement in text. That assumes the class exists and fires, and a check NAMED after a principle is not the same as a check that catches a violation of it. scripts/principle-coverage.py decides it by measurement. It builds a body violating one principle and nothing else, publishes it past the gate, and reports whether the gate blocked it. The clean baseline is a positive control: if it does not pass, every result is uninterpretable and the script says so — the first version of this measurement reported 9/9 enforced on a baseline that was itself blocked. Result: 3 of 9 enforced, 6 exist only as prose, including items 7, 8 and 9, which are the three most recently learned. Cutting those would delete the only place the rule exists, so the cut is not made. Item 2 is the subtle one and is recorded as such: check 8 keys on a provenance marker being present anywhere in the body, so a verdict reached by reading passes as long as an artifact sits nearby. The check is a proxy for the rule. What the measurement is actually good for is the inverse of the proposal — it names the six checks worth building, in priority order, and it can be re-run after each one to show the number move. `node --test` 61/61; gate-controls 11/11; the output quoted in skill.md is reproduced by running the script.
The rule is that a validation surface does not restate CI status: "tests are green at head <sha>" hands the reviewer their own Checks tab back and carries no information. The implementation matched a bare `actions/runs/N`, which is a different thing — five of its six branches described a CLAIM about CI, and one described a URL. So a link to a specific run and job whose log holds the figure being reported was refused, and that is precisely what evidence-run.yml exists to produce: "move the measurement to CI, where the run URL is the capture". The package forbade its own flagship output, and a real published body was blocked by it. A run link is now a violation only when it carries restatement language with it. The distinction is whether the sentence asserts a status the Checks tab already shows, or points at an execution whose output the Checks tab does not. scripts/ci-citation-controls.py holds the line, with five restatements that must be caught and three citations that must be allowed. It imports CI_RESTATEMENT from the hook rather than restating it, because a control that tests its own copy of a pattern passes forever while the real one drifts. before: 3 of 8 misclassified — every citation a false positive after: 0 of 8 the same controls against the old pattern still catch the citations, so the control has power rather than passing by construction gate-controls 11/11; replaying real published bodies now shows zero regressions, where the CI rule previously blocked one; principle-coverage unchanged at 3/9; `node --test` 61/61.
… into trial/refresh
…ying-test' into trial/refresh # Conflicts: # domains/pr-workflow/skills/evidence/hooks/pr-evidence-gate.py # domains/pr-workflow/skills/evidence/references/evidence-catalog.md
# Conflicts: # tools/install # tools/skill-schema.mjs
`tools/check-bundle-contract.sh` parses the array with a single-line `sed` pattern, so splitting it across lines made the contract check fail to parse it rather than fail a comparison — the check reported that it could not read the declaration and exited 2. The comment moves above the declaration.
`hooks/` was added here to `BUNDLE_DIRS` and to the installer's copy list, but `jongsun/add/pr-validate-falsifying-test` already does both in `edf65a3`, along with `tools/check-bundle-contract.sh` to assert the two lists agree. Two branches making the same change is how one of them ends up reverted by a merge resolution nobody reads. So this branch keeps only the part that is its own: refusing to overwrite an installed copy that has local changes. The guard iterates whatever `BUNDLE_DIRS` holds, so it covers `hooks/` for free once that branch lands. Dropping it also removes a defect this branch would otherwise carry into main. `check-bundle-contract.sh` reads the array with a single-line `sed`, and the multi-line form here made it fail to PARSE rather than fail a comparison — it exited 2 reporting that it could not read the declaration, which is a check that has stopped checking rather than one that found something. Controls re-run after narrowing, since the guard now iterates a different set: clean install 0, no-op re-install 0, source moves ahead 0 and propagates, local edit 1 with the sha unchanged, `--force` 0 and discards. 61/61.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Caution
DO NOT MERGE. This branch is a merge of fourteen open PRs, kept on
MetaMask/skillsso thewhole set can be installed and exercised as one thing. It has no content of its own — every line
in it belongs to one of the PRs below and should be reviewed there. It will be deleted once they
land.
Why this exists
No single PR installs into a working state.
mms-evidencealone routes to engines that live ineight different domains, so checking out its branch gets you an orchestrator naming skills that
are not installed. Reproducing the full set by hand means a fourteen-way merge against two
remotes, which is not a reasonable thing to ask of someone who wants to spend twenty minutes
trying it.
What is merged
39 skills across 13 PRs, plus one CI change.
--no-ffon top ofmain.jongsun/add/pr-validate-falsifying-testevidencejongsun/add/audit-skillsfalsifiers-first,unmeasured-join,instrument-check,scope-of-search,coverage-partition,selection-audit,silent-failure,unintended-breakage,red-on-base,distinguishing-observation,observability-gapjongsun/add/lane-graphslane-graphsjongsun/add/debug-orchestratordebugjongsun/add/memory-leak-hunt-skill-v2memory-leakjongsun/add/race-condition-proofrace-condition-reprojongsun/add/security-domainsupply-chain-audit,lavamoat-policyjongsun/add/privacy-egress-diligenceprivacy-egress-diligenceadd/performance(fork)react-render-delta,selector-antipattern-scan,effect-antipattern-scan,extension-profiling,data-analysis,benchmark-designadd/analytics-tracing-cross-ref(fork)sentry-quota,sentry-mcp-queries,sentry-grafana-correlation,grafana-tempo-queries,performance-attribution,instrumentationadd/typescript-typing-skill(fork)tsc-blindspots,avoid-any,derive-types,decompose-large-files,migration-context-costadd/platform-skills(fork)extension-errors-debugging,extension-lifecycle-decouplingjongsun/add/agent-run-costagent-run-costjongsun/ci/ship-hooks-and-trigger-checkhooks/, adds the installed-command checkOne conflict along the way, in
.github/CODEOWNERS, resolved by keeping both the/domains/security/and/domains/stability/lines.Deliberately not merged: #79 (
mobile-visual-testing), which is someone else's open PR, andthe CI/infra PRs #46, #87, #93, #94, #95, #103, which change tooling rather than add skills. Say
so if you want any of them in.
Install
Add
--dry-runfirst to see the file list without writing. That installs 65 skills formetamask-extension(70 formetamask-mobile, 43 forcore) into.claude/skills/mms-*/,.cursor/rules/and.agents/skills/.--savewrites the selection to.skills.localso laterruns repeat it.
Four things that are easy to get wrong:
--domain.--domain pr-workflowlooks correct and silently dropsred-on-base,memory-leak,react-render-deltaand the rest, whilemms-evidencestillnames them. The skills span fifteen domains:
pr-workflow(12),performance(9),testing(8),
analytics(6),typescript(5),coding(5),web3-tools(4),perps(4),security(3),
stability(2),platform(2),general(2),ui(1),swaps(1),agentic(1).--maturity experimentalis required. Everything exceptlane-graphsisstable, and thedefault filter is
stable— so a default run installs 64 of 65 and does not report which one itskipped.
package.json;override with
--repoif there isn't one.scope: project. They install into<target>/.claude/skills/, not$HOME.--targetdefaults to the current directory, so run it from — or point it at — the repo youwant them in.
To remove them:
--prune-staleon a later sync, or delete themms-*directories.Usage
Two orchestrators, and the rest are engines and lenses they route to. Everything also stands
alone, triggered by name or by description match — you do not have to route by hand.
mms-evidencetakes a claim about a PR and produces something a reader can check withouttrusting the author:
It runs three ways: on a PR whose claim someone else made, in the inner loop against uncommitted
changes before a reviewer sees them, and on a symptom with no claim yet, where the hypothesis to
kill is your own.
mms-debugis the same shape pointed at a symptom rather than a claim.Representative engines:
/mms-red-on-basefor a regression test that must fail on the basecommit;
/mms-silent-failurefor paths that can fail without saying so;/mms-unmeasured-joinfor the inference step between two true facts that nobody instrumented;
/mms-tsc-blindspotsforwhat the type checker does not catch;
/mms-sentry-mcp-queriesand/mms-sentry-grafana-correlationfor reading production signal;/mms-supply-chain-auditand/mms-privacy-egress-diligencefor dependency and egress review.What it is for. The bar is not that a run reaches the right verdict on every concern. It is
that it surfaces concerns, falsifiers and avenues of inquiry that a human then judges. A run that
ends "inconclusive, and here is what would settle it" has done its job.
Known state
Six lint errors, all pre-existing on the source branches rather than introduced by the merge —
mainis clean under the same linter. Five are the same class, from the check #99 adds: adescriptionadvertises/namewhere the installer emitsmms-name. Those branches predate therule and only fail once combined with it.
performance/react-render-delta/react-render-delta, installed asmms-react-render-deltastability/memory-leakstability/race-condition-reprotypescript/tsc-blindspotspr-workflow/debugpr-workflow/evidenceprobes/is not an allowed directory besideskill.mdThey are fixable only on the PRs that own them. Fixing them here would put the fix on a branch
that is never merged and leave the real one broken — which is the failure mode #84 was already
caught by once, when four runner fixes landed on the copy CI executed and never reached the copy
under review.
Also carried here:
domains/pr-workflow/skills/evidence/scripts/duplicatesa standalone runners repository (
v0.1.0). Thatrepository is the source; the copies are pending removal and exist so the branch can run
standalone while it is reviewed.
evidence-runners/tools/check-duplicates.shcompares them.