diff --git a/.github/skills/azsdk-common-eval-authoring-skill/SKILL.md b/.github/skills/azsdk-common-eval-authoring-skill/SKILL.md new file mode 100644 index 000000000000..fd4475c8edfc --- /dev/null +++ b/.github/skills/azsdk-common-eval-authoring-skill/SKILL.md @@ -0,0 +1,40 @@ +--- +name: azsdk-common-eval-authoring-skill +description: 'Author and validate Vally evals for Agent Skills under .github/skills. WHEN: "write a skill eval", "add trigger eval", "test skill routing", "add anti-trigger tests", "create skill capability eval", "harden skill graders". DO NOT USE FOR: single MCP tool prompt-to-tool coverage (use azsdk-common-eval-authoring-tool), multi-tool or multi-turn scenarios (use azsdk-common-eval-authoring-workflow), authoring SKILL.md itself (use skill-authoring).' +license: MIT +metadata: + author: Microsoft + version: "1.0.0" + distribution: shared +compatibility: "copilot-chat, @microsoft/vally-cli 0.7.0" +--- + +# Skill Eval Authoring + +Author Vally evals that verify **one Agent Skill's** routing and behavior in isolation, under `.github/skills//evals/`. All the actual guidance — naming convention, placement, per-category requirements, glossary, grading patterns, grader catalog, anti-patterns, and worked examples — lives in one shared place so it never drifts across the three eval-authoring skills: the eval authoring guide at `eng/common/knowledge/eval-authoring/README.md`. This skill exists to route you there with skill-eval context already loaded, not to duplicate it. + +## Triggers + +USE FOR: write a skill eval, add trigger eval, test skill routing, add anti-trigger tests, create skill capability eval, harden skill graders +WHEN: "write a skill eval", "add trigger eval", "test skill routing", "add anti-trigger tests", "create skill capability eval", "harden skill graders" +DO NOT USE FOR: single MCP tool prompt-to-tool coverage (use azsdk-common-eval-authoring-tool), multi-tool or multi-turn scenarios (use azsdk-common-eval-authoring-workflow), authoring SKILL.md itself (use skill-authoring) + +## Steps + +1. Read the eval authoring guide (`eng/common/knowledge/eval-authoring/README.md`) Step 0 to find this repo's `vallyRoot`/`evalGlobs` for the skill tier, then the guide's "Skill" column throughout (naming, requirements, worked example). +2. Read the target `SKILL.md` — its `WHEN`/`DO NOT USE FOR` boundaries, invoked tools — and its existing `evals/`. +3. Write `eval.yaml` with routing stimuli (trigger + anti-trigger, `skill-invocation` grader) and capability stimuli together in the same file per the guide's naming convention and four-layer pattern. Split into additional `.eval.yaml` files only once a skill's coverage genuinely grows large. For a boundary prompt, mount and **require** the intended competing skill while disallowing the skill under test — an anti-trigger with no competing skill mounted trivially "passes" (guide anti-pattern A7). +4. Supply concrete identifiers (repo path, package name, anything needed to reach the call) in every stimulus prompt — a vague prompt fails `tool-calls` with zero recorded calls, which looks like a routing bug but is really a missing-context prompt bug. +5. Validate locally per the guide's "Running evals locally" section. Do not finish or open a PR until the focused local eval passes; inspect `eval-results.md` and `results.jsonl` on failure. + +## Rules + +- Cover both activation and non-activation; anti-triggers should target neighboring skills, not random unrelated prompts. +- Omit `turn` for conversation-wide assertions; pin a turn only when that turn owns the behavior unambiguously. +- Never copy secrets or production-write identifiers into fixtures. +- Update existing specs additively unless the requested contract changed. + +## References + +- Eval authoring guide: `eng/common/knowledge/eval-authoring/README.md` — naming, placement, requirements, glossary, graders, anti-patterns, worked examples, local commands. Shared with `azsdk-common-eval-authoring-tool`/`azsdk-common-eval-authoring-workflow` — update it there, not per-skill. +- Repository-local eval README/configuration discovered in Step 0 (if present) diff --git a/.github/skills/azsdk-common-eval-authoring-skill/evals/eval.yaml b/.github/skills/azsdk-common-eval-authoring-skill/evals/eval.yaml new file mode 100644 index 000000000000..4feb187322c3 --- /dev/null +++ b/.github/skills/azsdk-common-eval-authoring-skill/evals/eval.yaml @@ -0,0 +1,123 @@ +name: azsdk-common-eval-authoring-skill-eval +description: Routing and boundary tests for the azsdk-common-eval-authoring-skill skill +type: capability + +environment: azsdk-mcp-mock + +tags: + area: azsdk-common-eval-authoring-skill + type: ci-gate + +defaults: + runs: 1 + timeout: "120s" + model: claude-opus-4.6 + executor: copilot-sdk + +scoring: + threshold: 0.8 + +stimuli: + # Boundary anti-triggers need the two sibling eval-authoring skills mounted + # alongside this one, or "disallowed" is trivially satisfied because no + # better-fitting skill was ever available to the model. + - name: trigger-write-skill-eval + environment: &siblingSkills + skills: + - .. + - ../../azsdk-common-eval-authoring-tool + - ../../azsdk-common-eval-authoring-workflow + prompt: "write a robust eval for my Agent Skill" + graders: + - type: skill-invocation + config: + required: [azsdk-common-eval-authoring-skill] + disallowed: + [ + azsdk-common-eval-authoring-tool, + azsdk-common-eval-authoring-workflow, + ] + + - name: trigger-add-trigger-and-anti-trigger-tests + environment: *siblingSkills + prompt: "Add Vally trigger and anti-trigger routing tests for .github/skills/azsdk-common-generate-sdk-locally. Its eval file is .github/skills/azsdk-common-generate-sdk-locally/evals/eval.yaml and it handles local TypeSpec SDK generation." + graders: + - type: skill-invocation + config: + required: [azsdk-common-eval-authoring-skill] + disallowed: + [ + azsdk-common-eval-authoring-tool, + azsdk-common-eval-authoring-workflow, + ] + + - name: trigger-test-skill-routing + environment: *siblingSkills + prompt: "Test routing for .github/skills/azsdk-common-pipeline-analysis: it should activate for Azure SDK CI/pipeline diagnosis requests but not for requests to fix the failure. Add the appropriate Vally routing eval cases." + graders: + - type: skill-invocation + config: + required: [azsdk-common-eval-authoring-skill] + disallowed: + [ + azsdk-common-eval-authoring-tool, + azsdk-common-eval-authoring-workflow, + ] + + - name: trigger-harden-skill-graders + environment: *siblingSkills + prompt: "harden my skill capability eval so refusals cannot pass" + graders: + - type: skill-invocation + config: + required: [azsdk-common-eval-authoring-skill] + disallowed: + [ + azsdk-common-eval-authoring-tool, + azsdk-common-eval-authoring-workflow, + ] + + - name: anti-trigger-single-mcp-tool-eval + environment: *siblingSkills + prompt: "add a prompt-to-tool eval for azsdk_get_release_plan under evals/tools" + graders: + - type: skill-invocation + config: + required: [azsdk-common-eval-authoring-tool] + disallowed: + [ + azsdk-common-eval-authoring-skill, + azsdk-common-eval-authoring-workflow, + ] + + - name: anti-trigger-multi-tool-workflow + environment: *siblingSkills + prompt: "write a mock workflow eval that checks two MCP tools are called in order" + graders: + - type: skill-invocation + config: + required: [azsdk-common-eval-authoring-workflow] + disallowed: + [ + azsdk-common-eval-authoring-skill, + azsdk-common-eval-authoring-tool, + ] + + - name: anti-trigger-author-skill + environment: + skills: + - .. + - ../../azsdk-common-eval-authoring-tool + - ../../azsdk-common-eval-authoring-workflow + - ../../skill-authoring + prompt: "create a new Agent Skill and write its SKILL.md" + graders: + - type: skill-invocation + config: + required: [skill-authoring] + disallowed: + [ + azsdk-common-eval-authoring-skill, + azsdk-common-eval-authoring-tool, + azsdk-common-eval-authoring-workflow, + ] diff --git a/.github/skills/azsdk-common-eval-authoring-tool/SKILL.md b/.github/skills/azsdk-common-eval-authoring-tool/SKILL.md new file mode 100644 index 000000000000..94eb291fdbf4 --- /dev/null +++ b/.github/skills/azsdk-common-eval-authoring-tool/SKILL.md @@ -0,0 +1,41 @@ +--- +name: azsdk-common-eval-authoring-tool +description: 'Author and validate hermetic single-tool Vally evals under evals/tools. WHEN: "write a tool eval", "add prompt-to-tool coverage", "test MCP tool selection", "add tool catalog eval", "create single-tool scenario", "harden tool-call grader". DO NOT USE FOR: skill routing or capability evals (use azsdk-common-eval-authoring-skill), multi-tool, multi-turn, or live scenarios (use azsdk-common-eval-authoring-workflow).' +license: MIT +metadata: + author: Microsoft + version: "1.0.0" + distribution: shared +compatibility: "copilot-chat, @microsoft/vally-cli 0.7.0" +--- + +# Tool Eval Authoring + +Author Vally evals that verify **one MCP tool** is selected correctly for a given prompt, under `evals/tools/`. All the actual guidance — naming convention, placement, per-category requirements, glossary, grading patterns, grader catalog, anti-patterns, and worked examples — lives in one shared place so it never drifts across the three eval-authoring skills: the eval authoring guide at `eng/common/knowledge/eval-authoring/README.md`. This skill exists to route you there with tool-eval context already loaded, not to duplicate it. + +## Triggers + +USE FOR: write a tool eval, add prompt-to-tool coverage, test MCP tool selection, add tool catalog eval, create single-tool scenario, harden tool-call grader +WHEN: "write a tool eval", "add prompt-to-tool coverage", "test MCP tool selection", "add tool catalog eval", "create single-tool scenario", "harden tool-call grader" +DO NOT USE FOR: skill routing or capability evals (use azsdk-common-eval-authoring-skill), multi-tool, multi-turn, or live scenarios (use azsdk-common-eval-authoring-workflow) + +## Steps + +1. Read the eval authoring guide (`eng/common/knowledge/eval-authoring/README.md`) Step 0 to find this repo's `vallyRoot`/`evalGlobs` for the tool tier, then the guide's "Tool" column throughout (naming, requirements, worked example). +2. Confirm the scenario expects one primary tool. If success requires orchestration, conversation state, or live services, use `azsdk-common-eval-authoring-workflow` instead. +3. Add stimuli to the matching `prompt-to-tool-.eval.yaml`; create a separate file only when it needs fixtures or outcome grading. +4. Use realistic, concrete prompts with multiple natural phrasings and collision cases that disallow the nearest competing tool. Make `tool-calls` the primary signal; avoid the anti-patterns in the guide (vacuous keyword-only grading, missing `scoring.threshold`). +5. Keep the unit tier hermetic (`environment: azsdk-mcp-mock`, `tags.tier: unit`, established area tag). Avoid git worktrees and production writes. +6. Validate locally per the guide's "Running evals locally" section, building the mock MCP first. Do not finish or open a PR until it passes; inspect recorded tool names and arguments on failure. + +## Rules + +- Use exact MCP tool names and assert forbidden alternatives where ambiguity exists. +- Do not force a tool through unnatural prompt instructions unless direct invocation is the contract being tested. +- Keep fixture paths relative to the eval file and fixture data minimal and non-secret. +- Preserve existing namespace coverage and add new stimuli instead of duplicating files. + +## References + +- Eval authoring guide: `eng/common/knowledge/eval-authoring/README.md` — naming, placement, requirements, glossary, graders, anti-patterns, worked examples, local commands. Shared with `azsdk-common-eval-authoring-skill`/`azsdk-common-eval-authoring-workflow` — update it there, not per-skill. +- Repository-local eval README/configuration discovered in Step 0 (if present) diff --git a/.github/skills/azsdk-common-eval-authoring-tool/evals/eval.yaml b/.github/skills/azsdk-common-eval-authoring-tool/evals/eval.yaml new file mode 100644 index 000000000000..cbd9848874be --- /dev/null +++ b/.github/skills/azsdk-common-eval-authoring-tool/evals/eval.yaml @@ -0,0 +1,117 @@ +name: azsdk-common-eval-authoring-tool-eval +description: Routing and boundary tests for the azsdk-common-eval-authoring-tool skill +type: capability + +environment: azsdk-mcp-mock + +tags: + area: azsdk-common-eval-authoring-tool + type: ci-gate + +defaults: + runs: 1 + timeout: "120s" + model: claude-opus-4.6 + executor: copilot-sdk + +scoring: + threshold: 0.8 + +stimuli: + # Boundary anti-triggers need the two sibling eval-authoring skills mounted + # alongside this one, or "disallowed" is trivially satisfied because no + # better-fitting skill was ever available to the model. + - name: trigger-write-tool-eval + environment: &siblingSkills + skills: + - .. + - ../../azsdk-common-eval-authoring-skill + - ../../azsdk-common-eval-authoring-workflow + prompt: "write a Vally eval for a single Azure SDK MCP tool" + graders: + - type: skill-invocation + config: + required: [azsdk-common-eval-authoring-tool] + disallowed: + [ + azsdk-common-eval-authoring-skill, + azsdk-common-eval-authoring-workflow, + ] + + - name: trigger-add-prompt-to-tool-coverage + environment: *siblingSkills + prompt: "add prompt-to-tool coverage for azsdk_get_release_plan" + graders: + - type: skill-invocation + config: + required: [azsdk-common-eval-authoring-tool] + disallowed: + [ + azsdk-common-eval-authoring-skill, + azsdk-common-eval-authoring-workflow, + ] + + - name: trigger-test-mcp-tool-selection + environment: *siblingSkills + prompt: "test whether user prompts select the correct MCP tool" + graders: + - type: skill-invocation + config: + required: [azsdk-common-eval-authoring-tool] + disallowed: + [ + azsdk-common-eval-authoring-skill, + azsdk-common-eval-authoring-workflow, + ] + + - name: trigger-harden-tool-call-grader + environment: *siblingSkills + prompt: "harden this single-tool eval against a competing tool call" + graders: + - type: skill-invocation + config: + required: [azsdk-common-eval-authoring-tool] + disallowed: + [ + azsdk-common-eval-authoring-skill, + azsdk-common-eval-authoring-workflow, + ] + + - name: anti-trigger-skill-routing + environment: *siblingSkills + prompt: "add trigger and anti-trigger tests to verify my Agent Skill activates for the right prompts" + graders: + - type: skill-invocation + config: + required: [azsdk-common-eval-authoring-skill] + disallowed: + [ + azsdk-common-eval-authoring-tool, + azsdk-common-eval-authoring-workflow, + ] + + - name: anti-trigger-multi-tool-workflow + environment: *siblingSkills + prompt: "create an end-to-end eval for a sequence of three MCP tools" + graders: + - type: skill-invocation + config: + required: [azsdk-common-eval-authoring-workflow] + disallowed: + [ + azsdk-common-eval-authoring-skill, + azsdk-common-eval-authoring-tool, + ] + + - name: anti-trigger-live-scenario + environment: *siblingSkills + prompt: "add a nightly live eval that writes to Azure DevOps" + graders: + - type: skill-invocation + config: + required: [azsdk-common-eval-authoring-workflow] + disallowed: + [ + azsdk-common-eval-authoring-skill, + azsdk-common-eval-authoring-tool, + ] diff --git a/.github/skills/azsdk-common-eval-authoring-workflow/SKILL.md b/.github/skills/azsdk-common-eval-authoring-workflow/SKILL.md new file mode 100644 index 000000000000..8c6e3467c731 --- /dev/null +++ b/.github/skills/azsdk-common-eval-authoring-workflow/SKILL.md @@ -0,0 +1,41 @@ +--- +name: azsdk-common-eval-authoring-workflow +description: 'Author and validate multi-tool, multi-turn, mock, and live Vally scenarios under evals/workflows. WHEN: "write a workflow eval", "add end-to-end eval", "test tool sequence", "create multi-turn eval", "add mock workflow scenario", "add live eval". DO NOT USE FOR: per-skill routing/capability evals (use azsdk-common-eval-authoring-skill), isolated prompt-to-tool tests (use azsdk-common-eval-authoring-tool).' +license: MIT +metadata: + author: Microsoft + version: "1.0.0" + distribution: shared +compatibility: "copilot-chat, @microsoft/vally-cli 0.7.0" +--- + +# Workflow Eval Authoring + +Author Vally evals that verify **multi-skill, multi-tool, or multi-turn** orchestration, state, and outcomes across an entire agent conversation, under `evals/workflows/`. All the actual guidance — naming convention, placement, per-category requirements, glossary, grading patterns, grader catalog, anti-patterns, and worked examples — lives in one shared place so it never drifts across the three eval-authoring skills: the eval authoring guide at `eng/common/knowledge/eval-authoring/README.md`. This skill exists to route you there with workflow-eval context already loaded, not to duplicate it. + +## Triggers + +USE FOR: write a workflow eval, add end-to-end eval, test tool sequence, create multi-turn eval, add mock workflow scenario, add live eval +WHEN: "write a workflow eval", "add end-to-end eval", "test tool sequence", "create multi-turn eval", "add mock workflow scenario", "add live eval" +DO NOT USE FOR: per-skill routing/capability evals (use azsdk-common-eval-authoring-skill), isolated prompt-to-tool tests (use azsdk-common-eval-authoring-tool) + +## Steps + +1. Read the eval authoring guide (`eng/common/knowledge/eval-authoring/README.md`) Step 0 to find this repo's `vallyRoot`/`evalGlobs` for the workflow tier, then the guide's "Workflow" column throughout (naming, requirements, worked example). +2. Use `evals/workflows/mock/` by default. Choose `live/` only when the behavior cannot be represented by the mock MCP; document writes, authentication, cleanup, and nightly-only execution. +3. Model one user goal per stimulus. Use `turns` only when conversation state matters; otherwise keep a single prompt. Mount every candidate skill explicitly, provide minimal file/git fixtures, and bound turns, tokens, workers, and timeout. +4. Combine process and outcome graders per the guide's four-layer pattern and grader catalog: required/disallowed skills and tools, ordering where essential, files/commands, response quality. Avoid overfitting to incidental call sequences and the guide's anti-patterns (e.g. boundary anti-triggers with no competing skill mounted). +5. Scope graders to `turn` only when that turn owns the assertion unambiguously; otherwise grade the full conversation. +6. Validate locally per the guide's "Running evals locally" section, building the matching MCP and priming git fixtures when required. Do not finish or open a PR until it passes; inspect every turn and tool call on failure. + +## Rules + +- Keep mock workflows hermetic and repeatable; use fake IDs and canned responses. +- Never run a live write scenario without the documented test area and safety environment. +- Preserve relative path invariants for `environment.skills`, `files`, and `git.source`. +- Use weights summing to `1.0` and a meaningful threshold when combining grader types. + +## References + +- Eval authoring guide: `eng/common/knowledge/eval-authoring/README.md` — naming, placement, requirements, glossary, graders, anti-patterns, worked examples, local commands. Shared with `azsdk-common-eval-authoring-skill`/`azsdk-common-eval-authoring-tool` — update it there, not per-skill. +- Repository-local eval README/configuration discovered in Step 0 (if present) diff --git a/.github/skills/azsdk-common-eval-authoring-workflow/evals/eval.yaml b/.github/skills/azsdk-common-eval-authoring-workflow/evals/eval.yaml new file mode 100644 index 000000000000..3bb1abc9509f --- /dev/null +++ b/.github/skills/azsdk-common-eval-authoring-workflow/evals/eval.yaml @@ -0,0 +1,117 @@ +name: azsdk-common-eval-authoring-workflow-eval +description: Routing and boundary tests for the azsdk-common-eval-authoring-workflow skill +type: capability + +environment: azsdk-mcp-mock + +tags: + area: azsdk-common-eval-authoring-workflow + type: ci-gate + +defaults: + runs: 1 + timeout: "120s" + model: claude-opus-4.6 + executor: copilot-sdk + +scoring: + threshold: 0.8 + +stimuli: + # Boundary anti-triggers need the two sibling eval-authoring skills mounted + # alongside this one, or "disallowed" is trivially satisfied because no + # better-fitting skill was ever available to the model. + - name: trigger-write-workflow-eval + environment: &siblingSkills + skills: + - .. + - ../../azsdk-common-eval-authoring-skill + - ../../azsdk-common-eval-authoring-tool + prompt: "write a Vally workflow eval for a multi-tool Azure SDK scenario" + graders: + - type: skill-invocation + config: + required: [azsdk-common-eval-authoring-workflow] + disallowed: + [ + azsdk-common-eval-authoring-skill, + azsdk-common-eval-authoring-tool, + ] + + - name: trigger-add-end-to-end-eval + environment: *siblingSkills + prompt: "add an end-to-end eval that verifies the whole agent workflow" + graders: + - type: skill-invocation + config: + required: [azsdk-common-eval-authoring-workflow] + disallowed: + [ + azsdk-common-eval-authoring-skill, + azsdk-common-eval-authoring-tool, + ] + + - name: trigger-test-tool-sequence + environment: *siblingSkills + prompt: "create a mock eval that checks several MCP tools are orchestrated correctly" + graders: + - type: skill-invocation + config: + required: [azsdk-common-eval-authoring-workflow] + disallowed: + [ + azsdk-common-eval-authoring-skill, + azsdk-common-eval-authoring-tool, + ] + + - name: trigger-create-multi-turn-eval + environment: *siblingSkills + prompt: "create a multi-turn conversation eval that reuses context from the first turn" + graders: + - type: skill-invocation + config: + required: [azsdk-common-eval-authoring-workflow] + disallowed: + [ + azsdk-common-eval-authoring-skill, + azsdk-common-eval-authoring-tool, + ] + + - name: trigger-add-live-eval + environment: *siblingSkills + prompt: "add a nightly live workflow eval for behavior the mock server cannot represent" + graders: + - type: skill-invocation + config: + required: [azsdk-common-eval-authoring-workflow] + disallowed: + [ + azsdk-common-eval-authoring-skill, + azsdk-common-eval-authoring-tool, + ] + + - name: anti-trigger-single-tool-selection + environment: *siblingSkills + prompt: "add two prompt-to-tool cases for one MCP tool under evals/tools" + graders: + - type: skill-invocation + config: + required: [azsdk-common-eval-authoring-tool] + disallowed: + [ + azsdk-common-eval-authoring-skill, + azsdk-common-eval-authoring-workflow, + ] + + - name: anti-trigger-skill-routing + environment: *siblingSkills + prompt: "add trigger and anti-trigger tests for my skill's routing" + graders: + - type: skill-invocation + config: + required: [azsdk-common-eval-authoring-skill] + disallowed: + [ + azsdk-common-eval-authoring-tool, + azsdk-common-eval-authoring-workflow, + ]