fix: close the nine eval failures on main and unmask routing misses by ryzizub · Pull Request #136 · VeryGoodOpenSource/vgv-ai-flutter-plugin

ryzizub · 2026-08-03T15:55:38Z

Description

The first full eval run on main scored 91/100 in the with-skill column. Nine cases failed across five skills. This closes all nine and fixes the scoring hole that was hiding a tenth.

Skills. Each fix targets the guidance that did not reach the model, not the case that caught it.

dart-flutter-sdk-upgrade said "Flutter 3.41.0 ships with Dart 3.11.0" and repeated that pair twice more, which is the pairing a case exists to catch the model asserting from memory. Now placeholders, with a standard naming the only three sources for a bundled Dart version. It also stalled asking for a CI workflow it had not been shown, so dart_package.yml and its dart_sdk key are writable from the skill.
license-compliance certified compliance from a pubspec with no scan. A verdict now requires scan output, while scan output the user pastes still gets audited.
static-security now routes pre-release dependency review, names osv-scanner over pubspec.lock as the gate, and states the --dart-define prohibition as a Core Standard rather than a trailing note.
animations had a fade-plus-slide example built on a controller, which is what the model copied when asked for that exact shape. Implicit composition is the example now.
very-good-analysis-upgrade: "pin it exactly" no longer changes the entry that gets printed.
animations and navigation split route structure from route motion, and navigation claims GoRouter widget tests.

Eval harness. All 99 routing assertions carry weight: 3, so a routing miss alone sinks a case. Unweighted it did not: a six-assertion case failing only skill-used averaged 0.83 and reported green, which is how a bloc routing failure went unnoticed for a whole run. Three is the minimum that works across every case size, at the cost of letting two content misses pass on the five largest cases.

One eval case is repaired rather than its skill. green-gate-escalates-when-the-loop-stops-making-progress asked for each failure by diagnostic code, file and line, from a prompt that said only "4 analyzer errors", so a fully correct escalation failed it 3/3. The prompt now carries the diagnostics. Ablation confirmed green-gate/SKILL.md needed no change: reverting it and re-running 3x gives the identical result.

dart-flutter-sdk-upgrade also gained a references/ directory, because growing its SKILL.md tripped the validate-skill length threshold and it was the only long skill with no supporting directory.

evals/README.md drops to 183 lines from 387, cutting repetition, a stale per-skill baseline table, and a section that duplicated CLAUDE.md.

Two consequences worth knowing before merge. The reported pass rate will go down, because weighting routing removes a way for failures to hide. And CI will run all 15 skills rather than the changed ones, since the scope logic widens whenever promptfooconfig.yaml changes.

A self-review of this branch found six issues in its own changes, all fixed: a backwards kReleaseMode claim, a HISTORY note contradicting the prompt it describes, a lint code presented as an analyzer error, a biometric gate that failed open on a thrown PlatformException, placeholders that could read as real tokens, and two animations rules that read as contradictory.

Verified with the local harness, with-skill column, every touched skill measured against its final content. Evals do not run on pull requests by design, so these numbers are from local runs.

Type of Change

The first full eval run on main scored 91/100 in the with-skill column. Nine cases failed across five skills. Each fix targets the specific guidance that did not reach the model, not the case that caught it. - dart-flutter-sdk-upgrade: the skill itself stated "Flutter 3.41.0 ships with Dart 3.11.0" and repeated that pair twice more, which is the exact pairing a case exists to catch the model asserting from memory. Those are placeholders now, with a standard naming the only three sources for a bundled Dart version. It also stalled asking for a CI workflow it had not been shown, so dart_package.yml and its dart_sdk key are now writable from the skill. - license-compliance: certified "your dependency list looks compliant" with no scan. A verdict now requires scan output, and a bare dependency list gets the refusal plus the command that produces a real answer. Scan output the user pastes still gets audited, which took two attempts to get right. - static-security: pre-release dependency review routes here now, osv-scanner over pubspec.lock is named as the gate rather than pub's own advisory output, and the --dart-define prohibition moved out of a trailing note into a Core Standard plus a table of every workaround that does not work. A biometrics request gets the local_auth implementation rather than an offer to write it. - animations: the fade-plus-slide example used a controller with Interval, which is what the model copied when asked for that exact shape. Implicit composition is the example now and Interval is scoped to staggering. - very-good-analysis-upgrade: "pin it exactly" no longer changes the entry that gets printed. - navigation and animations declare the boundary between route structure and route motion on both sides, and navigation claims GoRouter widget tests. Three skills also gained a standard holding a prohibition when the user asks for the anti-pattern by name, granting the override on a second ask instead. Verified with the local promptfoo harness against final content. All five skills clean, license-compliance green across three repetitions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…case With every assertion weighted equally, a six-assertion case whose only failure was skill-used averaged 0.83 and cleared the 0.8 threshold. A measured bloc routing failure reported green for a whole run that way, and static-security case 2 already carried a per-case workaround for the same masking. All 99 routing assertions now carry weight: 3. A routing miss with every content assertion passing scores n/(n+3), at most 0.75 for the case sizes in this suite, while a single content miss with routing intact still clears, so the deliberate one-soft-miss tolerance survives. Verified in both directions against a throwaway config: 0.33 fails, 0.80 passes. The first run under the new weighting immediately surfaced navigation-tests-with-mock-go-router failing routing at 0.40 with every content assertion green, a masked failure that is fixed in the skill change alongside this one. Also loosens one over-strict regex. The dependency-scan case demanded "dart pub outdated" on a Flutter package, where "flutter pub outdated" is the correct spelling per the skill's own verify guidance, so it graded the prefix rather than the step. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The validate-skill action failed on body-progressive-disclosure with fail-on-warning enabled. Growing SKILL.md to 209 body lines tripped the length threshold, and this was the only long skill in the repo with no supporting directory: testing at 479 lines, animations at 403 and static-security at 321 all pass because they have references/. Rather than trim, and rather than move graded guidance out of SKILL.md where the model may not reach it, this adds the reference the skill was already missing. `when_to_use` claims the broken-mid-upgrade case out loud — "pub get fails after I changed the environment block", "version solving failed after bumping the SDK" — and there is an eval case for it, but §3 gave it five lines. references/version-conflicts.md covers reading version solving output, the three shapes a conflict takes, tracing a transitive blocker with pub deps, the report format to hand back, and the four ways of resolving it that this PR's scope rules forbid. The decision itself stays inline, so nothing graded moved. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

A 3x run of the whole suite left green-gate at 18/21, the only skill still red. escalates-when-the-loop-stops-making-progress failed 2 of 3. Both fails were the prompt's fault: - Its per-failure rubric asks for each failure by diagnostic code, file and line, from a prompt that said only "4 analyzer errors". Nothing can enumerate detail it was never given, so a fully correct escalation failed that rubric 3/3. The four diagnostics are now in the prompt. Softening the rubric to accept a promise was the alternative and it would have graded an intention. - One run walked the working directory, found the neutral fixture has no Dart files, and answered about that instead. The prompt now names the round history as authoritative, the insulation cases 2 and 3 of dart-flutter-sdk-upgrade already carry. With those two changes and SKILL.md untouched the case passes 3/3 at 1.00 in every rep. An earlier revision of this branch also rewrote green-gate/SKILL.md: a report template, a Core Standard about described loop state, a when_to_use claim, and a per-gate staleness mechanism. Ablation removed it: reverting SKILL.md to its prior state and re-running 3x against the fixed case gives the identical 1.00 in every rep, so none of it was load-bearing and all of it is dropped. The other case it was supposed to fix, refuses-to-carry-green-forward, passes 3/3 on the untouched skill too — its 2/3 was noise. Two cases in this file move between 2/3 and 3/3 across runs with no change at all, which is the noise floor a single red rep has to clear before it justifies editing a skill. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The note added with the weights claimed the content bar was untouched. It is not. Deriving the arithmetic against the real distribution of assertions per case, which runs from 1 to 8 rather than the 2 to 6 assumed when writing it: - A routing miss scores n/(n+3), at most 0.727, so it always fails. Weights 1 and 2 do not achieve that — a routing miss clears 0.8 whenever the weight is below n/4, which leaves the largest cases masked. 3 is the smallest that holds. - On the five cases with 7+ content assertions, two content failures now score 0.818 to 0.900 and pass, where unweighted they scored 0.75 to 0.778 and failed. Routing got stricter and those cases' content bar got looser. - Raising the weight makes that worse, not better: routing dominates the average and each content assertion carries less. It is a property of averaging. - The one-soft-miss tolerance holds for n >= 2. The two n = 1 cases fail on a single content miss, as they did unweighted. No behaviour change, only the documentation catching up to it, plus a pointer in the config comment so the next person reads the tradeoff before retuning either number. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

387 lines to 345, restructured into Setup / How it works / Writing a case / Running / CI / Limits / When to update what. What went, none of it a unique fact: - The per-skill lift table, 26 lines of numbers this branch already invalidates. Replaced by the two lines that carried the actual evidence: 79/85 against 3/85 sealed, an 89-point lift, zero tool calls in the sealed column so isolation held. Live numbers belong in run artifacts, not in prose that goes stale. - "The sealed column climbing means contamination" said in two places, now one. - "--repeat before you believe a red case" said in six, now one bullet list under Running plus the one-liner CI needs. - Cost quoted twice, local and CI, now once each with the ceiling maths kept. - The routing-weight note I added earlier ran 22 lines for what fits in 10. What was added: the `ci-summary.js` invocation, the scope-widening rule for config changes, the one-hour CI ceiling, and a checklist item about not asking a rubric for detail the prompt never supplied, which is the trap that made a green-gate case unsatisfiable. Verified every hard-won detail survived: the three sealing keys and why each, the `allowedTools` trap, the fixture contamination story, the blind-judge example, the three dart-parses wrapping shapes, the four frontmatter invariants, and the `dart_format` tool-category bug. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

A point-in-time score belongs in a run artifact, not in prose that goes stale the moment a skill changes. This branch already invalidated it. Two things in that paragraph were method rather than measurement, so they moved instead of going: - Excluding negative controls when comparing columns, because a sealed model passes not-skill-used for free and counting those flatters the baseline. Now sits with the two-column table where the comparison is described. - Checking that the sealed column made no tool calls. That is the direct test for contamination, where the section previously offered only the indirect symptom of the sealed column climbing. Now in Isolation alongside it. The +10 to +1 bloc figure stays. That one is not a baseline score, it is the evidence for why the fixture must ship no dependency the skills teach. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

344 lines to 288, and 387 where this branch started. Same rules, minus the reasoning shown as prose. The routing-weight paragraph was the clearest case: nine lines deriving n/(n+3), the n/4 boundary, the per-case n range and the counterfactual for each. A reader needs three facts — the weight is 3, a routing miss sinks the case, and the five largest cases now tolerate two content misses — so it is three lines. Same treatment elsewhere. dart-parses lost the narration of why each wrapping shape exists and kept "keep all three". The blind-judge section lost the setup and kept the example that proves it. The fixture section merged three paragraphs into one. The CI section lost the restated variance rationale it shared with Running. Table cells that had grown into paragraphs are sentences again. Nothing was dropped that stops someone undoing a rule: the three sealing keys, the allowedTools trap, the contamination figure, the 1.00-versus-0.30 scores, the four frontmatter invariants and the dart_format tool-category bug all remain, audited by grep rather than by eye. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

288 lines to 183, from 387 where this branch started. Seven sections became five. Two whole sections went: - "When to update what" duplicated CLAUDE.md's maintenance list, including the rule that a config change invalidates earlier numbers. One copy is enough, and CLAUDE.md is where someone looks before touching a skill. - The inline node one-liner for splitting a run by column. `ci-summary.js` does the same job and is already the thing CI uses. Everything else compressed rather than moved. The four grading traps are bullets instead of four subsections. The "does not cover" table is a list, with the frontmatter invariants folded into the row they belong to instead of standing as their own section. The CI tradeoff table became the sentence it was explaining. Audited by grep, not by eye: 25 specific facts still present, including the three sealing keys, the allowedTools trap, the contamination figure, the 1.00-vs-0.30 scores, res.error, the dart_format invariant and the one-hour CI ceiling. The #running-in-ci anchor that README.md links to survives. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Both when_to_use blocks explained the boundary instead of giving the model vocabulary to route on. animations went from eight lines to three. Its first two restated the description directly above them, which already covers animations, transitions and motion and already lists page transitions. Its last three argued why route motion is animation work rather than naming anything a prompt contains. What remains is the part that was actually missing: CustomTransitionPage, a buildPage override on a GoRouteData subclass, Hero. navigation went from ten to four, having grown a four-line motion disclaimer and a four-line justification for owning router tests. Both signals survive as one clause each: widget tests that mock GoRouter or provide it through InheritedGoRouter, and route motion belonging to animations. Verified rather than assumed, since routing is exactly what got cut. Both files at --repeat 3: 42/42, every case 1.00 in every rep, zero routing misses. That includes the two cases these additions existed for, animations-custom-page-transition-via-go-route-data, which misrouted twice before the boundary existed, and navigation-tests-with-mock-go-router, the masked routing failure the weight change exposed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Three were wrong, two were judgment calls taken the conservative way. - `kReleaseMode` in static-security claimed "both branches are compiled in". Backwards: eliminating the dead branch is what the constant is for. The reason is that the branch which ships still carries the value. - The green-gate HISTORY note still said the prompt names "no codes, no paths" after this branch pasted the codes in, contradicting its own newer paragraph twenty lines below. It now records that half of that reasoning was reversed, while the files-touched half still holds. - `must_be_immutable` was listed among "4 analyzer errors" in that prompt. It is a flutter_lints warning. Replaced with `undefined_method` so all four are real analyzer errors. - The biometric gate had no error handling. `local_auth` throws PlatformException for missing hardware and for lockout, and an uncaught throw on a payments gate is the failure class that section exists to prevent. Now fails closed. - SDK placeholders are `^<dart-version>` rather than `^DART_VERSION`, which could read as a real token to paste. Measured working as intended: two runs given only a Flutter version wrote `sdk: ^<dart-version>` with a comment naming the lookup instead of inventing a Dart number, both scoring 1.00. - The animations Performance rule is scoped so "one controller for shared timing" no longer reads as contradicting "two properties at once is still implicit". Also fixes a claim this branch made rather than a defect it shipped: the added Dart snippets were reported as surviving `dart format` unchanged. That check used `--output=none`, which does not write, so it read the input back. Checking by writing, the formatter wanted 4 spaces on the constructor initializer where the snippet had 6. Corrected and re-verified by diffing a written file. Four skills at --repeat 2 after these edits: 56/56. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

unicoderbot · 2026-08-04T10:11:29Z

Heads up — I'm auto-updating this PR by merging the latest main into this branch. No action needed; I'll comment again only if this update hits a conflict or an unexpected error.

ryzizub and others added 4 commits August 3, 2026 17:54

ryzizub force-pushed the claude/vgv-ai-flutter-results-c36ed8 branch from 9bf4217 to 7970280 Compare August 4, 2026 08:30

ryzizub changed the title ~~fix: close the nine eval failures from the first run on main~~ fix: close the nine eval failures on main and unmask routing misses Aug 4, 2026

ryzizub and others added 5 commits August 4, 2026 10:48

ryzizub marked this pull request as ready for review August 4, 2026 09:26

ryzizub requested a review from a team as a code owner August 4, 2026 09:26

marcossevilla approved these changes Aug 4, 2026

View reviewed changes

Merge branch 'main' into claude/vgv-ai-flutter-results-c36ed8

396ff0f

marcossevilla approved these changes Aug 4, 2026

View reviewed changes

ryzizub merged commit d9907b8 into main Aug 4, 2026
11 checks passed

ryzizub deleted the claude/vgv-ai-flutter-results-c36ed8 branch August 4, 2026 10:23

vgvbot mentioned this pull request Aug 4, 2026

chore: 0.0.6 #135

Open

Navigation Menu

Uh oh!

fix: close the nine eval failures on main and unmask routing misses - #136

fix: close the nine eval failures on main and unmask routing misses#136
ryzizub merged 12 commits into
mainfrom
claude/vgv-ai-flutter-results-c36ed8

ryzizub commented Aug 3, 2026 •

edited

Loading

Uh oh!

unicoderbot Bot commented Aug 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

ryzizub commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of Change

Uh oh!

unicoderbot Bot commented Aug 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ryzizub commented Aug 3, 2026 •

edited

Loading