feat(PF-4363): add pf-assist agent and eval - #131
Conversation
📝 WalkthroughWalkthroughAdds the Changespf-assist plugin docs and eval cases
Estimated code review effort: 3 (Moderate) | ~25 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Warning Review ran into problems🔥 ProblemsLinked repositories: Your configuration references 1 linked repositories, but your current plan allows 0. Analyzed ``, skipped Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (4)
plugins/code-review/skills/pf-standards-check/SKILL.md (4)
1-95: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsumer-facing skill should have an eval.
Per path instructions, consumer-facing skills are expected to have an
eval/<skill-name>/eval.yaml. This skill lives incode-review(a consumer plugin), notpf-workshop. Consider adding an eval that proves the orchestrator's unique value — e.g., testing that it correctly gates onpackage.json, sequences all six sub-skills, and formats consolidated output.As per path instructions: "Consumer-facing skills are expected to have an eval.yaml in eval//. If this PR adds or significantly modifies a consumer skill and no corresponding eval exists, flag it as a nitpick reminder."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/code-review/skills/pf-standards-check/SKILL.md` around lines 1 - 95, The pf-standards-check consumer-facing skill is missing the required eval coverage, so add an eval under eval/pf-standards-check/eval.yaml for this orchestrator in code-review. Make the eval verify the gate check on package.json, the sequencing of all six sub-skills, and the consolidated report formatting produced by the pf-standards-check workflow.Source: Path instructions
92-92: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAdd language specifier to fenced code block.
The CLAUDE.md integration example lacks a language specifier. Add
textormarkdown.+```text
After code review or implementation workflows, run /pf-standards-check before collecting feedback.As per path instructions referencing CONTRIBUTING-SKILLS.md, keep documentation concise and AI-friendly. <details> <summary>🤖 Prompt for AI Agents</summary>Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.In
@plugins/code-review/skills/pf-standards-check/SKILL.mdat line 92, The
fenced example in SKILL.md is missing a language specifier, so update the
documentation block to use a tagged fence such as text or markdown. Locate the
CLAUDE.md integration example in the SKILL.md content and keep the surrounding
guidance concise and AI-friendly per CONTRIBUTING-SKILLS.md.</details> <!-- cr-comment:v1:517827ecfc60d72437851d04 --> _Source: Path instructions_ --- `2-4`: _📐 Maintainability & Code Quality_ | _🔵 Trivial_ | _💤 Low value_ **Consider a stronger action verb in the description.** "Check" is a weak verb per the skill description formula. Prefer "Audit" or "Validate" to front-load capability more clearly. ```diff -description: Check PatternFly code against all validation dimensions — component structure, imports, design tokens, colors, design compliance, and migration readiness. Use when PatternFly component code has been written, modified, or reviewed. +description: Audit PatternFly code against all validation dimensions — component structure, imports, design tokens, colors, design compliance, and migration readiness. Use when PatternFly component code has been written, modified, or reviewed.As per coding guidelines, descriptions should start with a strong action verb.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/code-review/skills/pf-standards-check/SKILL.md` around lines 2 - 4, The skill description in pf-standards-check uses the weak verb “Check”; update the top-level description to start with a stronger action verb such as “Audit” or “Validate” while preserving the rest of the scope, so the skill name and description read as a clear capability statement.Source: Coding guidelines
64-64: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAdd language specifier to fenced code block.
The output format example lacks a language specifier. Add
textormarkdownto satisfy markdownlint.+```text
PatternFly Standards Check Results
As per path instructions referencing CONTRIBUTING-SKILLS.md, keep documentation concise and AI-friendly. <details> <summary>🤖 Prompt for AI Agents</summary>Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.In
@plugins/code-review/skills/pf-standards-check/SKILL.mdat line 64, The
fenced code block in the output format example is missing a language specifier,
which should be added to satisfy markdownlint. Update the example in SKILL.md so
the fenced block around “PatternFly Standards Check Results” uses a specifier
like text or markdown, and keep the surrounding documentation concise and
AI-friendly as required by CONTRIBUTING-SKILLS.md.</details> <!-- cr-comment:v1:c83e9d9cc1a81851434cf8d7 --> _Source: Path instructions_ </blockquote></details> </blockquote></details> <details> <summary>🤖 Prompt for all review comments with AI agents</summary>Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.Nitpick comments:
In@plugins/code-review/skills/pf-standards-check/SKILL.md:
- Around line 1-95: The pf-standards-check consumer-facing skill is missing the
required eval coverage, so add an eval under eval/pf-standards-check/eval.yaml
for this orchestrator in code-review. Make the eval verify the gate check on
package.json, the sequencing of all six sub-skills, and the consolidated report
formatting produced by the pf-standards-check workflow.- Line 92: The fenced example in SKILL.md is missing a language specifier, so
update the documentation block to use a tagged fence such as text or markdown.
Locate the CLAUDE.md integration example in the SKILL.md content and keep the
surrounding guidance concise and AI-friendly per CONTRIBUTING-SKILLS.md.- Around line 2-4: The skill description in pf-standards-check uses the weak
verb “Check”; update the top-level description to start with a stronger action
verb such as “Audit” or “Validate” while preserving the rest of the scope, so
the skill name and description read as a clear capability statement.- Line 64: The fenced code block in the output format example is missing a
language specifier, which should be added to satisfy markdownlint. Update the
example in SKILL.md so the fenced block around “PatternFly Standards Check
Results” uses a specifier like text or markdown, and keep the surrounding
documentation concise and AI-friendly as required by CONTRIBUTING-SKILLS.md.</details> --- <details> <summary>ℹ️ Review info</summary> <details> <summary>⚙️ Run configuration</summary> **Configuration used**: Path: .coderabbit.yaml **Review profile**: CHILL **Plan**: Pro **Run ID**: `99d5b137-271b-484e-b184-183151cd52ad` </details> <details> <summary>📥 Commits</summary> Reviewing files that changed from the base of the PR and between 8e69cb557ad385a67b1afa5951ddc7f5f1afe322 and 8e23fb03bda991075fe63fc7f8be7fe043f90c0d. </details> <details> <summary>📒 Files selected for processing (6)</summary> * `CONTRIBUTING-SKILLS.md` * `PLUGINS.md` * `README.md` * `plugins/code-review/README.md` * `plugins/code-review/skills/pf-standards-check/SKILL.md` * `plugins/react/README.md` </details> </details> <!-- This is an auto-generated comment by CodeRabbit for review status -->
jpuzz0
changed the title
Convert pf-assist from skill to always-on agent. The routing knowledge (gate check, 4 context tables, context detection signals) is "always follow" knowledge that should activate contextually in any project with @patternfly/* dependencies — not require explicit invocation. Narrows .gitignore from eval/ to eval/runs/ so eval definitions are tracked while run artifacts stay ignored.
jpuzz0
changed the title
jpuzz0
changed the title
There was a problem hiding this comment.
🧹 Nitpick comments (1)
eval/pf-assist/cases/pf-validation-routing/src/components/StatusCard.tsx (1)
11-16: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRaw hex colors instead of PF tokens — presumably intentional bait for
/pf-raw-colors-scanand/pf-token-auditorrouting in this validation-routing eval case, so leaving as-is.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@eval/pf-assist/cases/pf-validation-routing/src/components/StatusCard.tsx` around lines 11 - 16, No code change is needed here; the raw hex colors in StatusCard are intentional for this validation-routing case, so leave the Card backgroundColor and span color styling unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@eval/pf-assist/cases/pf-validation-routing/src/components/StatusCard.tsx`:
- Around line 11-16: No code change is needed here; the raw hex colors in
StatusCard are intentional for this validation-routing case, so leave the Card
backgroundColor and span color styling unchanged.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: a6dc08f9-f48c-4286-ad14-b441c057d9c6
📒 Files selected for processing (13)
eval/pf-assist/cases/non-pf-gate-check/annotations.yamleval/pf-assist/cases/non-pf-gate-check/input.yamleval/pf-assist/cases/non-pf-gate-check/package.jsoneval/pf-assist/cases/non-pf-gate-check/src/components/UserCard.tsxeval/pf-assist/cases/pf-testing-gap/annotations.yamleval/pf-assist/cases/pf-testing-gap/input.yamleval/pf-assist/cases/pf-testing-gap/package.jsoneval/pf-assist/cases/pf-testing-gap/src/components/AlertBanner.tsxeval/pf-assist/cases/pf-validation-routing/annotations.yamleval/pf-assist/cases/pf-validation-routing/input.yamleval/pf-assist/cases/pf-validation-routing/package.jsoneval/pf-assist/cases/pf-validation-routing/src/components/StatusCard.tsxeval/pf-assist/eval.yaml
✅ Files skipped from review due to trivial changes (8)
- eval/pf-assist/cases/pf-validation-routing/input.yaml
- eval/pf-assist/cases/pf-testing-gap/input.yaml
- eval/pf-assist/cases/pf-validation-routing/package.json
- eval/pf-assist/cases/non-pf-gate-check/annotations.yaml
- eval/pf-assist/cases/pf-validation-routing/annotations.yaml
- eval/pf-assist/cases/non-pf-gate-check/src/components/UserCard.tsx
- eval/pf-assist/cases/pf-testing-gap/package.json
- eval/pf-assist/eval.yaml
Tests agent behavioral influence using skill: "" + plugin_dirs — no harness extension needed. Three cases: PF validation routing, non-PF gate check, and PF testing gap. Three judges verify sub-skill routing, gate check accuracy, and actionable next steps.
1 participant
Summary
pf-assistas an agent in thecode-reviewplugin — always-on routing knowledge that makes every PF consumer skill contextually available to any@patternfly/*project@patternfly/*inpackage.jsonand exits immediately on non-PF projectsskill: ""+plugin_dirs— tests behavioral influence (does the agent surface specific sub-skill names?) with 3 test cases and 3 judgesagentsfield in plugin.json for bothreactandcode-reviewplugins — required for marketplace installs to discover agents (auto-discovery only works with direct--plugin-dirinstalls)Skills routed contextually
/pf-component-structure,/pf-import-checker,/pf-raw-colors-scan,/pf-token-auditor,/pf-compliance-checker,/pf-class-migration-scanner/pf-unit-test-generator/pf-project-scaffolder/pf-figma-icon-finder,/pf-design-mode,/pf-design-comments,/pf-ai-experience-patternsEval coverage
pf-validation-routingroutes_to_subskillsnon-pf-gate-checkgate_skip_non_pfpf-testing-gaproutes_to_subskillsCloses: PF-4363