feat: add functional evals for 7 phase-1 skills - #23
Merged
Conversation
Adds evals.json + input fixtures under evals/ for 7 skills: applying-coding-baseline, autofixing-and-escalating, bumping-version, building-shared-vocabulary, guarding-agent-directives, managing-git-workflow, improving-architecture. Each skill has 2 test prompts with structural assertions plus DISCRIMINATING assertions targeting canonical wording, fixed question labels, severity classifications, or other artifacts only the skill is known to produce consistently. Iteration-1 results (28 subagent runs, deterministic grading): - Strong gain (delta >= 20%): guarding-agent-directives (+43%), applying-coding-baseline (+32%) - Moderate gain (5-20%): autofixing-and-escalating (+15%), building-shared-vocabulary (+14%) - No measurable gain: bumping-version, managing-git-workflow, improving-architecture (industry standards already known to baseline) Cost: with_skill averages +13% time/tokens (SKILL.md reading).
|
Warning Rate limit exceeded
To continue reviewing without waiting, purchase usage credits in the billing tab. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (9)
📝 WalkthroughWalkthroughAdds comprehensive evaluation configurations and fixture files across seven skills in the skill-set system: applying-coding-baseline, autofixing-and-escalating, building-shared-vocabulary, bumping-version, guarding-agent-directives, improving-architecture, and managing-git-workflow. Each skill receives an evals.json with scenario definitions and supporting fixture files (source code, documentation, or test data). ChangesSkill Evaluation Configurations and Fixtures
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Warning Review ran into problems🔥 ProblemsGit: Failed to clone repository. Please run the Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🧹 Nitpick comments (4)
plugins/skill-set/skills/building-shared-vocabulary/evals/evals.json (1)
17-17: ⚡ Quick winConsider documenting the
DISCRIMINATING:prefix convention.The
"DISCRIMINATING:"prefix is a magic sentinel string with special semantics for the deterministic grader (it identifies higher-stakes assertions used to compute the with_skill vs without_skill delta). Since JSON has no comment syntax, its meaning is entirely opaque to anyone reading this file without prior context.A lightweight fix could be a sibling
README.mdin theevals/directory explaining the eval schema, or a"$schema"reference if a JSON Schema is ever added. At minimum, a note in the top-level eval layout docs would help future contributors authoring new evals.As per coding guidelines for
**/*.json: "Document configuration values in scripts and avoid 'voodoo constants' - explain why each configuration value was chosen."Also applies to: 33-33
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/skill-set/skills/building-shared-vocabulary/evals/evals.json` at line 17, Add explicit documentation for the magic sentinel "DISCRIMINATING:" used in evals.json by creating a short evals/README.md that explains the prefix semantics (what it signals to the deterministic grader, how it affects with_skill vs without_skill scoring, and examples like the CONTEXT.md note about 'auth'), and also add an optional top-level "$schema" reference in evals.json (or update the top-level eval layout docs) to point to a future JSON Schema; ensure the README and any schema/doc update clearly call out the "DISCRIMINATING:" sentinel and recommend contributors avoid undocumented voodoo constants.plugins/skill-set/skills/bumping-version/evals/evals.json (2)
9-13: ⚡ Quick winRelative
filespaths lack a documented root.Paths such as
"evals/files/patch-bump-package-json/package.json"are relative, but the root they resolve against (skill directory? repo root? theevals/directory itself?) is not declared in the schema or in any accompanying documentation. If the eval harness convention changes, these silently break.Consider adding a
base_dirfield or documenting the resolution rule in a siblingREADME.md. As per coding guidelines, configuration values should be explained rather than left as implicit voodoo constants.Also applies to: 30-34
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/skill-set/skills/bumping-version/evals/evals.json` around lines 9 - 13, The evals.json "files" array uses relative paths without a declared resolution root causing brittle behavior; update the evals.json schema/instance by adding a new "base_dir" (or "files_base") field alongside the existing "files" array in the same JSON object and set it to the intended root (e.g., "evals/" or repo root), then update any code that consumes this file to join base_dir with entries from "files"; alternatively, add a README.md next to evals.json documenting the resolution rule (mentioning the "files" array and the new "base_dir" key) so consumers and future maintainers know how paths are resolved.
7-7: ⚡ Quick win"outputs directory" is an undocumented implicit contract.
Both prompts instruct the agent to "Save the modified files to the outputs directory" / "Save modified files to the outputs directory", but
evals.jsonhas nooutputs_dirfield and no comment explaining where this directory is or how the harness resolves it. Any reader (or a future eval runner) has to guess the convention.As per coding guidelines, configuration values should be documented and voodoo constants avoided. Consider adding a top-level or per-eval
outputs_dirfield, or at minimum add a comment block (e.g., in an accompanyingREADME.mdfor the evals layout) documenting the convention.📋 Example schema addition
{ "skill_name": "bumping-version", + "outputs_dir": "evals/outputs", "evals": [Also applies to: 28-28
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/skill-set/skills/bumping-version/evals/evals.json` at line 7, The evals.json prompts reference an undocumented "outputs directory"; update the eval configuration to make this explicit by adding a top-level or per-eval outputs_dir field (or an outputs_dir key next to the "prompt" entry) in plugins/skill-set/skills/bumping-version/evals/evals.json and include a short comment or README entry describing how the harness resolves that path so callers don't rely on a hidden convention; ensure the outputs_dir value is used by the eval harness and documented next to the "prompt" key so future readers see the contract.plugins/skill-set/skills/improving-architecture/evals/evals.json (1)
26-28: 💤 Low valueConsider documenting DISCRIMINATING pattern rationale.
The DISCRIMINATING expectations encode important design decisions about canonical vocabulary (depth/shallow, seam/interface, deletion test, "does not reappear"). While the choices are contextually clear from the skill name and PR objectives, the coding guideline suggests documenting configuration values.
Since JSON lacks native comments, consider one of:
- Adding a
"rationale"field to DISCRIMINATING expectations in the schema- Documenting the vocabulary choices in a skill-level README or evaluation guide
- Accepting the format limitation, as the current design is reasonably self-documenting
This is a minor enhancement; the current structure is clear and functional.
As per coding guidelines: "Document configuration values in scripts and avoid 'voodoo constants' — explain why each configuration value was chosen" applies to JSON files, though JSON's lack of comment support creates a format constraint.
Also applies to: 47-49
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/skill-set/skills/improving-architecture/evals/evals.json` around lines 26 - 28, The DISCRIMINATING expectation entries currently use domain-specific constants (e.g., 'shallow'/'deep', 'seam'/'interface', "does not reappear") without documented rationale; add a short "rationale" string to each DISCRIMINATING expectation object explaining why those canonical vocabulary choices were made (or alternatively create a skill-level README/evaluation guide that documents the same vocabulary and the deletion/reappearance rule), ensuring entries referencing candidates.md explicitly mention the expected canonical terms and the deletion-test intent so reviewers and automated checks understand the configuration.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@plugins/skill-set/skills/autofixing-and-escalating/evals/files/mixed-lint-output/lint-report.txt`:
- Around line 3-5: The lint-report contains two incorrect location annotations:
update the entry for getDisplayName in src/user.js (currently "src/user.js:8:1")
to the correct position where the function is defined ("src/user.js:14:1"
referencing getDisplayName), and fix the getOrderTotal entry in src/order.js
(currently "src/order.js:25:1") to the actual start line of getOrderTotal
("src/order.js:18:1"); ensure the report lines reference the correct file and
function names (getDisplayName, getOrderTotal) so AMBIGUOUS items point to real
locations.
In
`@plugins/skill-set/skills/bumping-version/evals/files/minor-bump-plugin-json/CHANGELOG.md`:
- Around line 1-19: Add the standard "Keep a Changelog" preamble to CHANGELOG.md
by inserting the two-line header exactly as in the other fixture: "All notable
changes to this project will be documented in this file." and "The format is
based on [Keep a Changelog](https://keepachangelog.com/)." before the existing
release sections (preserve the existing entries for [1.5.2], [1.5.1], [1.5.0]);
ensure the bracket-link format matches exactly so the eval expecting the "Keep a
Changelog" reference will pass.
In
`@plugins/skill-set/skills/improving-architecture/evals/files/shallow-validators-cluster/src/orders/admin.ts`:
- Around line 6-21: The function adminCreateOrder is marked async but performs
only synchronous calls (checkInventory, checkPromotion) so the async keyword is
unnecessary and changes error-handling/return semantics; remove async from the
adminCreateOrder declaration and update its return type to { ok: boolean;
errors?: Record<string,string> } (not Promise<...>) so callers get a plain
synchronous result, or alternatively keep async but add a clear comment above
adminCreateOrder stating it is intentionally async for future I/O and why;
ensure you update any call sites if you remove the Promise return to match the
synchronous signature.
In
`@plugins/skill-set/skills/managing-git-workflow/evals/files/commit-with-conventional-style/setup.sh`:
- Around line 4-10: Add an explicit empty-workdir preflight in setup.sh before
running git init: check for any files/dirs (e.g. using ls -A or checking git
status if repo exists) and if non-empty print a clear error like "expected empty
directory for fixture" to stderr and exit non-zero; place this check immediately
before the git init -q -b main line so the script fails fast with a helpful
message rather than seeding a flaky repo.
- Around line 65-66: The current pattern reassigns module.exports to an object
with parse and normalizeNestedArrays and then sets exports.version, which
mutates a stale reference and won’t surface on the exported object; consolidate
into a single export so the version property is part of the exported object
(e.g. include version alongside parse and normalizeNestedArrays in the
module.exports assignment) or ensure you mutate the exact exported object
reference rather than using the stale exports alias; update the export around
the parse and normalizeNestedArrays symbols and the version value accordingly.
In
`@plugins/skill-set/skills/managing-git-workflow/evals/files/pr-title-and-body-generation/setup.sh`:
- Around line 4-10: The script assumes an empty working directory but doesn't
guard against being run inside an existing repo or non-empty dir; add a
preflight check before git init (and alongside set -euo pipefail) that fails
fast with a clear message if the cwd is non-empty or already a git repo (use git
rev-parse --is-inside-work-tree or test for existing .git and for any files
besides expected fixture markers), and provide a safe alternative in the message
(e.g., instruct to run in an empty directory, remove/backup files, or set a
FORCE_ENV var to override); ensure the failing path prints the helpful message
and exits non-zero.
---
Nitpick comments:
In `@plugins/skill-set/skills/building-shared-vocabulary/evals/evals.json`:
- Line 17: Add explicit documentation for the magic sentinel "DISCRIMINATING:"
used in evals.json by creating a short evals/README.md that explains the prefix
semantics (what it signals to the deterministic grader, how it affects
with_skill vs without_skill scoring, and examples like the CONTEXT.md note about
'auth'), and also add an optional top-level "$schema" reference in evals.json
(or update the top-level eval layout docs) to point to a future JSON Schema;
ensure the README and any schema/doc update clearly call out the
"DISCRIMINATING:" sentinel and recommend contributors avoid undocumented voodoo
constants.
In `@plugins/skill-set/skills/bumping-version/evals/evals.json`:
- Around line 9-13: The evals.json "files" array uses relative paths without a
declared resolution root causing brittle behavior; update the evals.json
schema/instance by adding a new "base_dir" (or "files_base") field alongside the
existing "files" array in the same JSON object and set it to the intended root
(e.g., "evals/" or repo root), then update any code that consumes this file to
join base_dir with entries from "files"; alternatively, add a README.md next to
evals.json documenting the resolution rule (mentioning the "files" array and the
new "base_dir" key) so consumers and future maintainers know how paths are
resolved.
- Line 7: The evals.json prompts reference an undocumented "outputs directory";
update the eval configuration to make this explicit by adding a top-level or
per-eval outputs_dir field (or an outputs_dir key next to the "prompt" entry) in
plugins/skill-set/skills/bumping-version/evals/evals.json and include a short
comment or README entry describing how the harness resolves that path so callers
don't rely on a hidden convention; ensure the outputs_dir value is used by the
eval harness and documented next to the "prompt" key so future readers see the
contract.
In `@plugins/skill-set/skills/improving-architecture/evals/evals.json`:
- Around line 26-28: The DISCRIMINATING expectation entries currently use
domain-specific constants (e.g., 'shallow'/'deep', 'seam'/'interface', "does not
reappear") without documented rationale; add a short "rationale" string to each
DISCRIMINATING expectation object explaining why those canonical vocabulary
choices were made (or alternatively create a skill-level README/evaluation guide
that documents the same vocabulary and the deletion/reappearance rule), ensuring
entries referencing candidates.md explicitly mention the expected canonical
terms and the deletion-test intent so reviewers and automated checks understand
the configuration.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: db4a674e-7607-4ad6-be71-f642044c5601
📒 Files selected for processing (34)
plugins/skill-set/skills/applying-coding-baseline/evals/evals.jsonplugins/skill-set/skills/applying-coding-baseline/evals/files/greenfield-no-claude-md/README.mdplugins/skill-set/skills/applying-coding-baseline/evals/files/normalize-paraphrased-rules/CLAUDE.mdplugins/skill-set/skills/autofixing-and-escalating/evals/evals.jsonplugins/skill-set/skills/autofixing-and-escalating/evals/files/mixed-lint-output/lint-report.txtplugins/skill-set/skills/autofixing-and-escalating/evals/files/mixed-lint-output/src/order.jsplugins/skill-set/skills/autofixing-and-escalating/evals/files/mixed-lint-output/src/user.jsplugins/skill-set/skills/autofixing-and-escalating/evals/files/mixed-pr-review-comments/review-comments.mdplugins/skill-set/skills/autofixing-and-escalating/evals/files/mixed-pr-review-comments/src/auth.jsplugins/skill-set/skills/building-shared-vocabulary/evals/evals.jsonplugins/skill-set/skills/building-shared-vocabulary/evals/files/adr-worthy-decision/README.mdplugins/skill-set/skills/building-shared-vocabulary/evals/files/first-domain-term/README.mdplugins/skill-set/skills/bumping-version/evals/evals.jsonplugins/skill-set/skills/bumping-version/evals/files/minor-bump-plugin-json/.claude-plugin/plugin.jsonplugins/skill-set/skills/bumping-version/evals/files/minor-bump-plugin-json/CHANGELOG.mdplugins/skill-set/skills/bumping-version/evals/files/minor-bump-plugin-json/recent-commits.txtplugins/skill-set/skills/bumping-version/evals/files/patch-bump-package-json/CHANGELOG.mdplugins/skill-set/skills/bumping-version/evals/files/patch-bump-package-json/package.jsonplugins/skill-set/skills/bumping-version/evals/files/patch-bump-package-json/recent-commits.txtplugins/skill-set/skills/guarding-agent-directives/evals/evals.jsonplugins/skill-set/skills/guarding-agent-directives/evals/files/accept-specific-rule/CLAUDE.mdplugins/skill-set/skills/guarding-agent-directives/evals/files/reject-vague-rule/CLAUDE.mdplugins/skill-set/skills/improving-architecture/evals/evals.jsonplugins/skill-set/skills/improving-architecture/evals/files/deletion-test-on-thin-wrapper/src/feature-a.tsplugins/skill-set/skills/improving-architecture/evals/files/deletion-test-on-thin-wrapper/src/feature-b.tsplugins/skill-set/skills/improving-architecture/evals/files/deletion-test-on-thin-wrapper/src/utils/string-helper.tsplugins/skill-set/skills/improving-architecture/evals/files/shallow-validators-cluster/src/orders/admin.tsplugins/skill-set/skills/improving-architecture/evals/files/shallow-validators-cluster/src/orders/credit-check.tsplugins/skill-set/skills/improving-architecture/evals/files/shallow-validators-cluster/src/orders/inventory-check.tsplugins/skill-set/skills/improving-architecture/evals/files/shallow-validators-cluster/src/orders/promotion-check.tsplugins/skill-set/skills/improving-architecture/evals/files/shallow-validators-cluster/src/orders/route.tsplugins/skill-set/skills/managing-git-workflow/evals/evals.jsonplugins/skill-set/skills/managing-git-workflow/evals/files/commit-with-conventional-style/setup.shplugins/skill-set/skills/managing-git-workflow/evals/files/pr-title-and-body-generation/setup.sh
Apply OBVIOUS fixture corrections and approved AMBIGUOUS items from CodeRabbit's review of the phase-1 evals PR. OBVIOUS (auto-applied): - mixed-lint-output/lint-report.txt: correct two stale line numbers (getDisplayName 8:1 -> 14:3, getOrderTotal 25:1 -> 18:3) - minor-bump-plugin-json/CHANGELOG.md: add Keep a Changelog preamble for parity with patch-bump fixture (eval expects bracket-link format) - commit-with-conventional-style/setup.sh: add empty-dir preflight guard - pr-title-and-body-generation/setup.sh: add empty-dir preflight guard AMBIGUOUS (escalated, user-approved): - shallow-validators-cluster/admin.ts: drop unnecessary async on adminCreateOrder (function body is fully synchronous) - commit-with-conventional-style/setup.sh: consolidate stale CommonJS exports into single module.exports object Nitpicks (3 of 4 user-approved): - Add plugins/skill-set/skills/EVALS.md documenting the eval schema: DISCRIMINATING: prefix semantics, files-path resolution rule, and the outputs/ directory convention - Add slim evals/README.md pointers in building-shared-vocabulary, bumping-version, and improving-architecture referencing EVALS.md - Skipped: per-expectation rationale field (CodeRabbit rated low value)
Owner
Author
|
@coderabbitai resolve Review feedback applied — cycle 1 (HEAD
|
| # | File | Fix |
|---|---|---|
| 1 | plugins/skill-set/skills/autofixing-and-escalating/evals/files/mixed-lint-output/lint-report.txt |
Corrected stale line numbers: getDisplayName 8:1 → 14:3, getOrderTotal 25:1 → 18:3 (verified against user.js / order.js). |
| 2 | plugins/skill-set/skills/bumping-version/evals/files/minor-bump-plugin-json/CHANGELOG.md |
Added Keep a Changelog preamble for parity with the patch-bump fixture. |
| 3 | plugins/skill-set/skills/managing-git-workflow/evals/files/commit-with-conventional-style/setup.sh |
Added empty-dir preflight guard before git init. |
| 4 | plugins/skill-set/skills/managing-git-workflow/evals/files/pr-title-and-body-generation/setup.sh |
Added empty-dir preflight guard before git init. |
Applied after discussion (AMBIGUOUS, 2 items, user-approved)
| # | File | Fix | Why ambiguous |
|---|---|---|---|
| 5 | plugins/skill-set/skills/improving-architecture/evals/files/shallow-validators-cluster/src/orders/admin.ts |
Dropped unnecessary async on adminCreateOrder; simplified return type. |
The eval grades on depth/shallow vocabulary, not async semantics — could be argued as intentional fixture noise. User approved CodeRabbit's exact diff. |
| 6 | plugins/skill-set/skills/managing-git-workflow/evals/files/commit-with-conventional-style/setup.sh (lines 65–66 of the staged change) |
Consolidated module.exports = {...}; exports.version = '...'; into a single export object. |
The fixture might intentionally simulate real-world stale-reference noise. User approved consolidation. The seed-history commit (chore(parser): expose version constant) is preserved unchanged in the seeded git log. |
Nitpicks (4 items, 3 user-approved)
| # | Decision |
|---|---|
Nitpick 1 — building-shared-vocabulary/evals/evals.json (DISCRIMINATING: prefix doc) |
Applied via canonical plugins/skill-set/skills/EVALS.md + slim per-skill evals/README.md pointer. |
Nitpick 2 — bumping-version/evals/evals.json (files path resolution rule) |
Applied via the same canonical EVALS.md (resolution rule is the skill's evals/ directory) + bumping-version/evals/README.md pointer. |
Nitpick 3 — bumping-version/evals/evals.json (outputs_dir convention) |
Applied via the same canonical EVALS.md. Schema field deferred — current change only documents the convention. |
Nitpick 4 — improving-architecture/evals/evals.json (per-expectation rationale field) |
Skipped. CodeRabbit itself rated this 💤 Low value; current DISCRIMINATING expectations are self-documenting through their assertion text. |
Statistics
- Total reviewed: 10 (6 inline + 4 nitpicks)
- Auto-applied (OBVIOUS): 4
- Discussed and applied (AMBIGUOUS + nitpicks): 5
- Skipped: 1 (nitpick 4, low value per the reviewer)
|
Tip For best results, initiate chat on the files or code changes.
(ﾉ◕ヮ◕)ﾉ*:･ﾟ✧ [resolve] |
Owner
Author
Cycle 3 — Verification PassHEAD: Scan results
Carry-over inline comments (3) — all verified already addressed
These comments still have Conclusion: All prior CodeRabbit feedback addressed. No commit, no push. |
This was referenced Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
evals/directory.DISCRIMINATINGassertions targeting canonical wording, fixed Q1–Q5 labels, OBVIOUS/AMBIGUOUS severity classes, deletion-test framing, and other artifacts only the skill is known to produce consistently.skills/<name>/evals/evals.json+evals/files/<eval-name>/...) that future skill iterations can extend.Iteration-1 results
Ran 28 parallel general-purpose subagents (7 skills × 2 evals × 2 configs) and graded with a deterministic per-skill grader. After adding
DISCRIMINATINGassertions, the with_skill vs without_skill differential becomes:Pattern. Skills that codify novel canonical wording or named procedures show clear gain. Skills that codify well-known industry standards (Keep a Changelog, Conventional Commits, Ousterhout deep modules) show no measurable gain — the baseline already knows them.
Cost. with_skill averages +13% time/tokens (SKILL.md + reference reading).
Changes
plugins/skill-set/skills/applying-coding-baseline/evals/— 2 evals, fixtures for greenfield CLAUDE.md and paraphrased-rule normalization.plugins/skill-set/skills/autofixing-and-escalating/evals/— 2 evals, fixtures for mixed lint output and PR review comments.plugins/skill-set/skills/bumping-version/evals/— 2 evals, fixtures for patch (package.json) and minor (plugin.json) bumps.plugins/skill-set/skills/building-shared-vocabulary/evals/— 2 evals, fixtures for first-term glossary and ADR-worthy decision.plugins/skill-set/skills/guarding-agent-directives/evals/— 2 evals, fixtures for vague-rule rejection and specific-rule acceptance.plugins/skill-set/skills/managing-git-workflow/evals/— 2 evals withsetup.shscripts that initialize fixture git repos for commit-message and PR-title generation.plugins/skill-set/skills/improving-architecture/evals/— 2 evals, fixtures for shallow-validator cluster and deletion-test on thin wrapper.Transient artifacts (subagent runs, deterministic grader, benchmarks, static HTML viewers) live under
.context/skill-evals/and are excluded from this PR.Test Plan
evals.jsonfiles parse as valid JSON in each of the 7 skills.evals/files/*/setup.shscripts (managing-git-workflow) run without error in a fresh empty directory..context/skill-evals/outputs reproduces the iteration-1 numbers above.evals/evals.json+evals/files/<eval-name>/...).Summary by CodeRabbit