diff --git a/.github/skills/azsdk-common-eval-authoring-skill/SKILL.md b/.github/skills/azsdk-common-eval-authoring-skill/SKILL.md deleted file mode 100644 index fd4475c8edfc..000000000000 --- a/.github/skills/azsdk-common-eval-authoring-skill/SKILL.md +++ /dev/null @@ -1,40 +0,0 @@ ---- -name: azsdk-common-eval-authoring-skill -description: 'Author and validate Vally evals for Agent Skills under .github/skills. WHEN: "write a skill eval", "add trigger eval", "test skill routing", "add anti-trigger tests", "create skill capability eval", "harden skill graders". DO NOT USE FOR: single MCP tool prompt-to-tool coverage (use azsdk-common-eval-authoring-tool), multi-tool or multi-turn scenarios (use azsdk-common-eval-authoring-workflow), authoring SKILL.md itself (use skill-authoring).' -license: MIT -metadata: - author: Microsoft - version: "1.0.0" - distribution: shared -compatibility: "copilot-chat, @microsoft/vally-cli 0.7.0" ---- - -# Skill Eval Authoring - -Author Vally evals that verify **one Agent Skill's** routing and behavior in isolation, under `.github/skills//evals/`. All the actual guidance — naming convention, placement, per-category requirements, glossary, grading patterns, grader catalog, anti-patterns, and worked examples — lives in one shared place so it never drifts across the three eval-authoring skills: the eval authoring guide at `eng/common/knowledge/eval-authoring/README.md`. This skill exists to route you there with skill-eval context already loaded, not to duplicate it. - -## Triggers - -USE FOR: write a skill eval, add trigger eval, test skill routing, add anti-trigger tests, create skill capability eval, harden skill graders -WHEN: "write a skill eval", "add trigger eval", "test skill routing", "add anti-trigger tests", "create skill capability eval", "harden skill graders" -DO NOT USE FOR: single MCP tool prompt-to-tool coverage (use azsdk-common-eval-authoring-tool), multi-tool or multi-turn scenarios (use azsdk-common-eval-authoring-workflow), authoring SKILL.md itself (use skill-authoring) - -## Steps - -1. Read the eval authoring guide (`eng/common/knowledge/eval-authoring/README.md`) Step 0 to find this repo's `vallyRoot`/`evalGlobs` for the skill tier, then the guide's "Skill" column throughout (naming, requirements, worked example). -2. Read the target `SKILL.md` — its `WHEN`/`DO NOT USE FOR` boundaries, invoked tools — and its existing `evals/`. -3. Write `eval.yaml` with routing stimuli (trigger + anti-trigger, `skill-invocation` grader) and capability stimuli together in the same file per the guide's naming convention and four-layer pattern. Split into additional `.eval.yaml` files only once a skill's coverage genuinely grows large. For a boundary prompt, mount and **require** the intended competing skill while disallowing the skill under test — an anti-trigger with no competing skill mounted trivially "passes" (guide anti-pattern A7). -4. Supply concrete identifiers (repo path, package name, anything needed to reach the call) in every stimulus prompt — a vague prompt fails `tool-calls` with zero recorded calls, which looks like a routing bug but is really a missing-context prompt bug. -5. Validate locally per the guide's "Running evals locally" section. Do not finish or open a PR until the focused local eval passes; inspect `eval-results.md` and `results.jsonl` on failure. - -## Rules - -- Cover both activation and non-activation; anti-triggers should target neighboring skills, not random unrelated prompts. -- Omit `turn` for conversation-wide assertions; pin a turn only when that turn owns the behavior unambiguously. -- Never copy secrets or production-write identifiers into fixtures. -- Update existing specs additively unless the requested contract changed. - -## References - -- Eval authoring guide: `eng/common/knowledge/eval-authoring/README.md` — naming, placement, requirements, glossary, graders, anti-patterns, worked examples, local commands. Shared with `azsdk-common-eval-authoring-tool`/`azsdk-common-eval-authoring-workflow` — update it there, not per-skill. -- Repository-local eval README/configuration discovered in Step 0 (if present) diff --git a/.github/skills/azsdk-common-eval-authoring-skill/evals/eval.yaml b/.github/skills/azsdk-common-eval-authoring-skill/evals/eval.yaml deleted file mode 100644 index 4feb187322c3..000000000000 --- a/.github/skills/azsdk-common-eval-authoring-skill/evals/eval.yaml +++ /dev/null @@ -1,123 +0,0 @@ -name: azsdk-common-eval-authoring-skill-eval -description: Routing and boundary tests for the azsdk-common-eval-authoring-skill skill -type: capability - -environment: azsdk-mcp-mock - -tags: - area: azsdk-common-eval-authoring-skill - type: ci-gate - -defaults: - runs: 1 - timeout: "120s" - model: claude-opus-4.6 - executor: copilot-sdk - -scoring: - threshold: 0.8 - -stimuli: - # Boundary anti-triggers need the two sibling eval-authoring skills mounted - # alongside this one, or "disallowed" is trivially satisfied because no - # better-fitting skill was ever available to the model. - - name: trigger-write-skill-eval - environment: &siblingSkills - skills: - - .. - - ../../azsdk-common-eval-authoring-tool - - ../../azsdk-common-eval-authoring-workflow - prompt: "write a robust eval for my Agent Skill" - graders: - - type: skill-invocation - config: - required: [azsdk-common-eval-authoring-skill] - disallowed: - [ - azsdk-common-eval-authoring-tool, - azsdk-common-eval-authoring-workflow, - ] - - - name: trigger-add-trigger-and-anti-trigger-tests - environment: *siblingSkills - prompt: "Add Vally trigger and anti-trigger routing tests for .github/skills/azsdk-common-generate-sdk-locally. Its eval file is .github/skills/azsdk-common-generate-sdk-locally/evals/eval.yaml and it handles local TypeSpec SDK generation." - graders: - - type: skill-invocation - config: - required: [azsdk-common-eval-authoring-skill] - disallowed: - [ - azsdk-common-eval-authoring-tool, - azsdk-common-eval-authoring-workflow, - ] - - - name: trigger-test-skill-routing - environment: *siblingSkills - prompt: "Test routing for .github/skills/azsdk-common-pipeline-analysis: it should activate for Azure SDK CI/pipeline diagnosis requests but not for requests to fix the failure. Add the appropriate Vally routing eval cases." - graders: - - type: skill-invocation - config: - required: [azsdk-common-eval-authoring-skill] - disallowed: - [ - azsdk-common-eval-authoring-tool, - azsdk-common-eval-authoring-workflow, - ] - - - name: trigger-harden-skill-graders - environment: *siblingSkills - prompt: "harden my skill capability eval so refusals cannot pass" - graders: - - type: skill-invocation - config: - required: [azsdk-common-eval-authoring-skill] - disallowed: - [ - azsdk-common-eval-authoring-tool, - azsdk-common-eval-authoring-workflow, - ] - - - name: anti-trigger-single-mcp-tool-eval - environment: *siblingSkills - prompt: "add a prompt-to-tool eval for azsdk_get_release_plan under evals/tools" - graders: - - type: skill-invocation - config: - required: [azsdk-common-eval-authoring-tool] - disallowed: - [ - azsdk-common-eval-authoring-skill, - azsdk-common-eval-authoring-workflow, - ] - - - name: anti-trigger-multi-tool-workflow - environment: *siblingSkills - prompt: "write a mock workflow eval that checks two MCP tools are called in order" - graders: - - type: skill-invocation - config: - required: [azsdk-common-eval-authoring-workflow] - disallowed: - [ - azsdk-common-eval-authoring-skill, - azsdk-common-eval-authoring-tool, - ] - - - name: anti-trigger-author-skill - environment: - skills: - - .. - - ../../azsdk-common-eval-authoring-tool - - ../../azsdk-common-eval-authoring-workflow - - ../../skill-authoring - prompt: "create a new Agent Skill and write its SKILL.md" - graders: - - type: skill-invocation - config: - required: [skill-authoring] - disallowed: - [ - azsdk-common-eval-authoring-skill, - azsdk-common-eval-authoring-tool, - azsdk-common-eval-authoring-workflow, - ] diff --git a/.github/skills/azsdk-common-eval-authoring-tool/SKILL.md b/.github/skills/azsdk-common-eval-authoring-tool/SKILL.md deleted file mode 100644 index 94eb291fdbf4..000000000000 --- a/.github/skills/azsdk-common-eval-authoring-tool/SKILL.md +++ /dev/null @@ -1,41 +0,0 @@ ---- -name: azsdk-common-eval-authoring-tool -description: 'Author and validate hermetic single-tool Vally evals under evals/tools. WHEN: "write a tool eval", "add prompt-to-tool coverage", "test MCP tool selection", "add tool catalog eval", "create single-tool scenario", "harden tool-call grader". DO NOT USE FOR: skill routing or capability evals (use azsdk-common-eval-authoring-skill), multi-tool, multi-turn, or live scenarios (use azsdk-common-eval-authoring-workflow).' -license: MIT -metadata: - author: Microsoft - version: "1.0.0" - distribution: shared -compatibility: "copilot-chat, @microsoft/vally-cli 0.7.0" ---- - -# Tool Eval Authoring - -Author Vally evals that verify **one MCP tool** is selected correctly for a given prompt, under `evals/tools/`. All the actual guidance — naming convention, placement, per-category requirements, glossary, grading patterns, grader catalog, anti-patterns, and worked examples — lives in one shared place so it never drifts across the three eval-authoring skills: the eval authoring guide at `eng/common/knowledge/eval-authoring/README.md`. This skill exists to route you there with tool-eval context already loaded, not to duplicate it. - -## Triggers - -USE FOR: write a tool eval, add prompt-to-tool coverage, test MCP tool selection, add tool catalog eval, create single-tool scenario, harden tool-call grader -WHEN: "write a tool eval", "add prompt-to-tool coverage", "test MCP tool selection", "add tool catalog eval", "create single-tool scenario", "harden tool-call grader" -DO NOT USE FOR: skill routing or capability evals (use azsdk-common-eval-authoring-skill), multi-tool, multi-turn, or live scenarios (use azsdk-common-eval-authoring-workflow) - -## Steps - -1. Read the eval authoring guide (`eng/common/knowledge/eval-authoring/README.md`) Step 0 to find this repo's `vallyRoot`/`evalGlobs` for the tool tier, then the guide's "Tool" column throughout (naming, requirements, worked example). -2. Confirm the scenario expects one primary tool. If success requires orchestration, conversation state, or live services, use `azsdk-common-eval-authoring-workflow` instead. -3. Add stimuli to the matching `prompt-to-tool-.eval.yaml`; create a separate file only when it needs fixtures or outcome grading. -4. Use realistic, concrete prompts with multiple natural phrasings and collision cases that disallow the nearest competing tool. Make `tool-calls` the primary signal; avoid the anti-patterns in the guide (vacuous keyword-only grading, missing `scoring.threshold`). -5. Keep the unit tier hermetic (`environment: azsdk-mcp-mock`, `tags.tier: unit`, established area tag). Avoid git worktrees and production writes. -6. Validate locally per the guide's "Running evals locally" section, building the mock MCP first. Do not finish or open a PR until it passes; inspect recorded tool names and arguments on failure. - -## Rules - -- Use exact MCP tool names and assert forbidden alternatives where ambiguity exists. -- Do not force a tool through unnatural prompt instructions unless direct invocation is the contract being tested. -- Keep fixture paths relative to the eval file and fixture data minimal and non-secret. -- Preserve existing namespace coverage and add new stimuli instead of duplicating files. - -## References - -- Eval authoring guide: `eng/common/knowledge/eval-authoring/README.md` — naming, placement, requirements, glossary, graders, anti-patterns, worked examples, local commands. Shared with `azsdk-common-eval-authoring-skill`/`azsdk-common-eval-authoring-workflow` — update it there, not per-skill. -- Repository-local eval README/configuration discovered in Step 0 (if present) diff --git a/.github/skills/azsdk-common-eval-authoring-tool/evals/eval.yaml b/.github/skills/azsdk-common-eval-authoring-tool/evals/eval.yaml deleted file mode 100644 index cbd9848874be..000000000000 --- a/.github/skills/azsdk-common-eval-authoring-tool/evals/eval.yaml +++ /dev/null @@ -1,117 +0,0 @@ -name: azsdk-common-eval-authoring-tool-eval -description: Routing and boundary tests for the azsdk-common-eval-authoring-tool skill -type: capability - -environment: azsdk-mcp-mock - -tags: - area: azsdk-common-eval-authoring-tool - type: ci-gate - -defaults: - runs: 1 - timeout: "120s" - model: claude-opus-4.6 - executor: copilot-sdk - -scoring: - threshold: 0.8 - -stimuli: - # Boundary anti-triggers need the two sibling eval-authoring skills mounted - # alongside this one, or "disallowed" is trivially satisfied because no - # better-fitting skill was ever available to the model. - - name: trigger-write-tool-eval - environment: &siblingSkills - skills: - - .. - - ../../azsdk-common-eval-authoring-skill - - ../../azsdk-common-eval-authoring-workflow - prompt: "write a Vally eval for a single Azure SDK MCP tool" - graders: - - type: skill-invocation - config: - required: [azsdk-common-eval-authoring-tool] - disallowed: - [ - azsdk-common-eval-authoring-skill, - azsdk-common-eval-authoring-workflow, - ] - - - name: trigger-add-prompt-to-tool-coverage - environment: *siblingSkills - prompt: "add prompt-to-tool coverage for azsdk_get_release_plan" - graders: - - type: skill-invocation - config: - required: [azsdk-common-eval-authoring-tool] - disallowed: - [ - azsdk-common-eval-authoring-skill, - azsdk-common-eval-authoring-workflow, - ] - - - name: trigger-test-mcp-tool-selection - environment: *siblingSkills - prompt: "test whether user prompts select the correct MCP tool" - graders: - - type: skill-invocation - config: - required: [azsdk-common-eval-authoring-tool] - disallowed: - [ - azsdk-common-eval-authoring-skill, - azsdk-common-eval-authoring-workflow, - ] - - - name: trigger-harden-tool-call-grader - environment: *siblingSkills - prompt: "harden this single-tool eval against a competing tool call" - graders: - - type: skill-invocation - config: - required: [azsdk-common-eval-authoring-tool] - disallowed: - [ - azsdk-common-eval-authoring-skill, - azsdk-common-eval-authoring-workflow, - ] - - - name: anti-trigger-skill-routing - environment: *siblingSkills - prompt: "add trigger and anti-trigger tests to verify my Agent Skill activates for the right prompts" - graders: - - type: skill-invocation - config: - required: [azsdk-common-eval-authoring-skill] - disallowed: - [ - azsdk-common-eval-authoring-tool, - azsdk-common-eval-authoring-workflow, - ] - - - name: anti-trigger-multi-tool-workflow - environment: *siblingSkills - prompt: "create an end-to-end eval for a sequence of three MCP tools" - graders: - - type: skill-invocation - config: - required: [azsdk-common-eval-authoring-workflow] - disallowed: - [ - azsdk-common-eval-authoring-skill, - azsdk-common-eval-authoring-tool, - ] - - - name: anti-trigger-live-scenario - environment: *siblingSkills - prompt: "add a nightly live eval that writes to Azure DevOps" - graders: - - type: skill-invocation - config: - required: [azsdk-common-eval-authoring-workflow] - disallowed: - [ - azsdk-common-eval-authoring-skill, - azsdk-common-eval-authoring-tool, - ] diff --git a/.github/skills/azsdk-common-eval-authoring-workflow/SKILL.md b/.github/skills/azsdk-common-eval-authoring-workflow/SKILL.md deleted file mode 100644 index 8c6e3467c731..000000000000 --- a/.github/skills/azsdk-common-eval-authoring-workflow/SKILL.md +++ /dev/null @@ -1,41 +0,0 @@ ---- -name: azsdk-common-eval-authoring-workflow -description: 'Author and validate multi-tool, multi-turn, mock, and live Vally scenarios under evals/workflows. WHEN: "write a workflow eval", "add end-to-end eval", "test tool sequence", "create multi-turn eval", "add mock workflow scenario", "add live eval". DO NOT USE FOR: per-skill routing/capability evals (use azsdk-common-eval-authoring-skill), isolated prompt-to-tool tests (use azsdk-common-eval-authoring-tool).' -license: MIT -metadata: - author: Microsoft - version: "1.0.0" - distribution: shared -compatibility: "copilot-chat, @microsoft/vally-cli 0.7.0" ---- - -# Workflow Eval Authoring - -Author Vally evals that verify **multi-skill, multi-tool, or multi-turn** orchestration, state, and outcomes across an entire agent conversation, under `evals/workflows/`. All the actual guidance — naming convention, placement, per-category requirements, glossary, grading patterns, grader catalog, anti-patterns, and worked examples — lives in one shared place so it never drifts across the three eval-authoring skills: the eval authoring guide at `eng/common/knowledge/eval-authoring/README.md`. This skill exists to route you there with workflow-eval context already loaded, not to duplicate it. - -## Triggers - -USE FOR: write a workflow eval, add end-to-end eval, test tool sequence, create multi-turn eval, add mock workflow scenario, add live eval -WHEN: "write a workflow eval", "add end-to-end eval", "test tool sequence", "create multi-turn eval", "add mock workflow scenario", "add live eval" -DO NOT USE FOR: per-skill routing/capability evals (use azsdk-common-eval-authoring-skill), isolated prompt-to-tool tests (use azsdk-common-eval-authoring-tool) - -## Steps - -1. Read the eval authoring guide (`eng/common/knowledge/eval-authoring/README.md`) Step 0 to find this repo's `vallyRoot`/`evalGlobs` for the workflow tier, then the guide's "Workflow" column throughout (naming, requirements, worked example). -2. Use `evals/workflows/mock/` by default. Choose `live/` only when the behavior cannot be represented by the mock MCP; document writes, authentication, cleanup, and nightly-only execution. -3. Model one user goal per stimulus. Use `turns` only when conversation state matters; otherwise keep a single prompt. Mount every candidate skill explicitly, provide minimal file/git fixtures, and bound turns, tokens, workers, and timeout. -4. Combine process and outcome graders per the guide's four-layer pattern and grader catalog: required/disallowed skills and tools, ordering where essential, files/commands, response quality. Avoid overfitting to incidental call sequences and the guide's anti-patterns (e.g. boundary anti-triggers with no competing skill mounted). -5. Scope graders to `turn` only when that turn owns the assertion unambiguously; otherwise grade the full conversation. -6. Validate locally per the guide's "Running evals locally" section, building the matching MCP and priming git fixtures when required. Do not finish or open a PR until it passes; inspect every turn and tool call on failure. - -## Rules - -- Keep mock workflows hermetic and repeatable; use fake IDs and canned responses. -- Never run a live write scenario without the documented test area and safety environment. -- Preserve relative path invariants for `environment.skills`, `files`, and `git.source`. -- Use weights summing to `1.0` and a meaningful threshold when combining grader types. - -## References - -- Eval authoring guide: `eng/common/knowledge/eval-authoring/README.md` — naming, placement, requirements, glossary, graders, anti-patterns, worked examples, local commands. Shared with `azsdk-common-eval-authoring-skill`/`azsdk-common-eval-authoring-tool` — update it there, not per-skill. -- Repository-local eval README/configuration discovered in Step 0 (if present) diff --git a/.github/skills/azsdk-common-eval-authoring-workflow/evals/eval.yaml b/.github/skills/azsdk-common-eval-authoring-workflow/evals/eval.yaml deleted file mode 100644 index 3bb1abc9509f..000000000000 --- a/.github/skills/azsdk-common-eval-authoring-workflow/evals/eval.yaml +++ /dev/null @@ -1,117 +0,0 @@ -name: azsdk-common-eval-authoring-workflow-eval -description: Routing and boundary tests for the azsdk-common-eval-authoring-workflow skill -type: capability - -environment: azsdk-mcp-mock - -tags: - area: azsdk-common-eval-authoring-workflow - type: ci-gate - -defaults: - runs: 1 - timeout: "120s" - model: claude-opus-4.6 - executor: copilot-sdk - -scoring: - threshold: 0.8 - -stimuli: - # Boundary anti-triggers need the two sibling eval-authoring skills mounted - # alongside this one, or "disallowed" is trivially satisfied because no - # better-fitting skill was ever available to the model. - - name: trigger-write-workflow-eval - environment: &siblingSkills - skills: - - .. - - ../../azsdk-common-eval-authoring-skill - - ../../azsdk-common-eval-authoring-tool - prompt: "write a Vally workflow eval for a multi-tool Azure SDK scenario" - graders: - - type: skill-invocation - config: - required: [azsdk-common-eval-authoring-workflow] - disallowed: - [ - azsdk-common-eval-authoring-skill, - azsdk-common-eval-authoring-tool, - ] - - - name: trigger-add-end-to-end-eval - environment: *siblingSkills - prompt: "add an end-to-end eval that verifies the whole agent workflow" - graders: - - type: skill-invocation - config: - required: [azsdk-common-eval-authoring-workflow] - disallowed: - [ - azsdk-common-eval-authoring-skill, - azsdk-common-eval-authoring-tool, - ] - - - name: trigger-test-tool-sequence - environment: *siblingSkills - prompt: "create a mock eval that checks several MCP tools are orchestrated correctly" - graders: - - type: skill-invocation - config: - required: [azsdk-common-eval-authoring-workflow] - disallowed: - [ - azsdk-common-eval-authoring-skill, - azsdk-common-eval-authoring-tool, - ] - - - name: trigger-create-multi-turn-eval - environment: *siblingSkills - prompt: "create a multi-turn conversation eval that reuses context from the first turn" - graders: - - type: skill-invocation - config: - required: [azsdk-common-eval-authoring-workflow] - disallowed: - [ - azsdk-common-eval-authoring-skill, - azsdk-common-eval-authoring-tool, - ] - - - name: trigger-add-live-eval - environment: *siblingSkills - prompt: "add a nightly live workflow eval for behavior the mock server cannot represent" - graders: - - type: skill-invocation - config: - required: [azsdk-common-eval-authoring-workflow] - disallowed: - [ - azsdk-common-eval-authoring-skill, - azsdk-common-eval-authoring-tool, - ] - - - name: anti-trigger-single-tool-selection - environment: *siblingSkills - prompt: "add two prompt-to-tool cases for one MCP tool under evals/tools" - graders: - - type: skill-invocation - config: - required: [azsdk-common-eval-authoring-tool] - disallowed: - [ - azsdk-common-eval-authoring-skill, - azsdk-common-eval-authoring-workflow, - ] - - - name: anti-trigger-skill-routing - environment: *siblingSkills - prompt: "add trigger and anti-trigger tests for my skill's routing" - graders: - - type: skill-invocation - config: - required: [azsdk-common-eval-authoring-skill] - disallowed: - [ - azsdk-common-eval-authoring-tool, - azsdk-common-eval-authoring-workflow, - ] diff --git a/eng/common/knowledge/eval-authoring/README.md b/eng/common/knowledge/eval-authoring/README.md deleted file mode 100644 index a0a3456cd7ff..000000000000 --- a/eng/common/knowledge/eval-authoring/README.md +++ /dev/null @@ -1,297 +0,0 @@ -# Eval authoring guide - -One guide for writing Vally evals in this repo, whatever you're testing: a single MCP **tool**, an Agent **skill**'s routing/behavior, or a multi-step **workflow**. Read this once. The three `azsdk-common-eval-authoring-*` skills exist only to route you here with the right category already identified — they don't duplicate this content. - -This doc lives under `eng/common/knowledge/` because it's synced as-is to every repo that consumes the shared eval pipeline. Everything here is grader- and process-level, not tied to one repo's file layout — where **your** repo's evals actually live is one lookup away (Step 0 below), not hardcoded in this guide. - -## Contents - -1. [Which category is this?](#1-which-category-is-this) -2. [Naming convention](#2-naming-convention) -3. [Category requirements](#3-category-requirements) -4. [Glossary](#4-glossary) -5. [Four-layer grading pattern](#5-four-layer-grading-pattern) -6. [Grader catalog](#6-grader-catalog) -7. [Anti-patterns](#7-anti-patterns) -8. [Customization guidance](#8-customization-guidance) -9. [Worked examples](#9-worked-examples) -10. [Running evals locally](#10-running-evals-locally) - -## Step 0: find out where evals live in *your* repo - -Before anything else: find the pipeline file that extends `eng/common/pipelines/templates/stages/archetype-eval.yml` for the category you're authoring, and read its `vallyRoot`/`evalGlobs`. Do not assume this repo's paths are universal. - -| Category | Azure SDK Tools example pipeline | Azure SDK Tools example `vallyRoot` | -| --- | --- | --- | -| Skill | `eng/common/pipelines/skill-eval.yml` | `.github/skills` | -| Tool / Workflow | `eng/common/pipelines/workflow-eval.yml`, `eng/common/pipelines/live-eval.yml` | `evals` | - -These are source-repository examples, not a contract for consuming repos. A repo that scopes skill evals under `.github/skills` may equally scope tool/workflow evals under `.github/skills/evals/{tools,workflows}` instead of a root-level `evals/` — always confirm against that repo's own pipeline file, never assume. - -## 1. Which category is this? - -```mermaid -flowchart TD - A[What are you testing?] --> B{One MCP tool's prompt-based\nselection, no orchestration?} - B -->|Yes| Tool[["Tool eval\nevals/tools/"]] - B -->|No| C{One skill's routing or\nbehavior, in isolation?} - C -->|Yes| Skill[["Skill eval\n.github/skills/<skill>/evals/"]] - C -->|No| D{Multiple skills/tools, multiple\nturns, or a full agent scenario?} - D -->|Yes| Workflow[["Workflow eval\nevals/workflows/{mock,live}/"]] -``` - -If you're unsure between skill and workflow: a **skill** eval has one target skill and checks its routing or behavior in isolation. Mount immediate competitors only for a boundary-routing stimulus. A **workflow** eval mounts every genuinely competing skill/tool together and checks orchestration across them (which one wins, in what order, across how many turns). - -## 2. Naming convention - -The point of a naming convention is that related files and results **sort and group together** — in a directory listing, in `vally eval` output, and when filtering by tag. - -### Files - -| Category | Filename pattern | Example | -| --- | --- | --- | -| Skill | `eval.yaml` (routing + capability stimuli together; split into `.eval.yaml` only once coverage grows large) | `.github/skills/my-skill/evals/eval.yaml` | -| Tool | `prompt-to-tool-.eval.yaml` (one per tool namespace/area) | `evals/tools/prompt-to-tool-pipeline.eval.yaml` | -| Workflow | `.eval.yaml` | `evals/workflows/mock/release-planner-workflows.eval.yaml` | - -### Stimuli (the `name:` field inside a file) - -- Routing stimuli: prefix `trigger-...` / `anti-trigger-...` so a scan of results instantly separates "should activate" from "should not." -- Everything else: a short kebab-case description of the scenario, e.g. `create-then-generate-conversation`. - -### Tags - -- `tags.area` must equal the skill name or tool-area exactly — this is what lets `--tag area=X` and tag-scoped suites (`release-plan`, `typespec`, `pipeline`, `github`, ...) find your file. -- `tags.type`/`tags.tier` use the small fixed vocabulary already in use (`ci-gate`, `unit`, ...) — match an existing sibling file, don't invent a new value. - -## 3. Category requirements - -| | Tool | Skill | Workflow | -| --- | --- | --- | --- | -| **Purpose** | Given a prompt, is the right MCP tool selected? | Given a prompt, does the right skill activate, and does it call the right tools? | Given a scenario (possibly multi-turn), does the right combination of skills/tools reach the right outcome? | -| **Location** | `evals/tools/` | `.github/skills//evals/` | `evals/workflows/{mock,live}/` | -| **Must have** | Realistic, concrete prompt (identifiers included, not left for the agent to discover); hermetic mock environment | Routing stimuli (trigger **and** anti-trigger, `skill-invocation` grader) alongside capability stimuli in `eval.yaml`; boundary anti-triggers mount **and require** a genuine competing skill | Every candidate skill mounted explicitly; bounded turns/tokens/timeout; choose the `vally eval --workers` value deliberately; mock by default, live only when the mock can't represent it | -| **Primary graders** | `tool-calls` | `skill-invocation` (+ `tool-calls`, `output-matches` for capability) | `skill-invocation` + `tool-calls` (ordering where it matters) + outcome graders | -| **Do NOT use for** | Orchestration, conversation state, live services → use Workflow | Single-tool selection → use Tool; multi-skill orchestration → use Workflow | Isolated single-tool or single-skill checks → use Tool/Skill instead | - -## 4. Glossary - -Plain-English definitions of the eval fields and CLI options you'll actually touch: - -| Term | Meaning | -| --- | --- | -| **stimulus** | One test case: a `prompt` (or `turns`) plus the `graders` that judge the response. A file is a list of stimuli. | -| **trajectory** | The full recorded sequence of tool calls and messages the agent produced for a stimulus. Graders inspect this, not just the final reply. | -| **turns** | An ordered list of prompts within one stimulus, for scenarios where a later request depends on earlier session context. Omit it and use a single `prompt` unless you specifically need that. | -| **timeout** | Max wall-clock time one stimulus run may take before it's failed as timed out. Live/orchestration-heavy stimuli need more headroom than a single tool call. | -| **model** | Which LLM executes the stimulus (e.g. `claude-opus-4.6`, `gpt-5.4`). Use a stronger model where judgment matters; a cheaper/faster one is fine for high-volume routing suites. | -| **executor** | The harness driving the model + tools for a stimulus. `copilot-sdk` is the real Copilot Chat SDK loop used across this repo's suites. | -| **environment** | A named MCP server + skill-mount configuration (e.g. `azsdk-mcp-mock` vs `azsdk-mcp-live`) that a stimulus runs against. | -| **scoring.threshold** | The minimum aggregate score (0–1) for an eval suite to pass. A suite can clear its threshold even if an individual stimulus fails. Always set it explicitly — repo convention: `0.8`. | -| **scoring.weights** | Optional per-grader weighting. When present, must sum to `1.0` or the spec fails to load. | -| **workers** | How many stimuli `vally eval` runs in parallel. Live/write-side suites should use `--workers 1` to avoid racing real side effects. | - -## 5. Four-layer grading pattern - -Every stimulus verifies a *contract*. These are the dimensions of that contract — use as many as apply, minimum routing + tool-use for any capability stimulus: - -1. **Routing** (`skill-invocation`) — did the right skill get loaded (and the wrong one not)? -2. **Tool-use** (`tool-calls`) — did the agent call the right MCP tool, and avoid the wrong one? `name` is matched as a regex against the trajectory; prefer bare tool names over server-prefixed forms. -3. **Output shape** (`output-matches`) — does the reply structurally address the request? Prefer a regex requiring two related concepts in proximity over a bare `output-contains` keyword — much harder to satisfy by accident. -4. **Judgment** (`prompt`) — anything the first three can't express. Use sparingly; LLM judges are expensive and non-deterministic. - -| Stimulus kind | Required layers | -| --- | --- | -| Capability — happy path | Routing + Tool-use, optionally Output-shape | -| Capability — multi-tool flow | Routing + Tool-use (multiple required entries, optionally ordered) | -| Negative — wrong topic | Routing disallowed + Tool-use disallowed | -| Boundary — sibling skill should win instead | Routing required for the sibling / disallowed for this skill; mount both together | -| Trigger | Routing required only | -| Anti-trigger | Routing disallowed only | - -## 6. Grader catalog - -| Grader | Purpose | Typical use | -| --- | --- | --- | -| `skill-invocation` | `required` / `disallowed` skills loaded during the run | Trigger, anti-trigger, boundary tests | -| `tool-calls` | `required` / `disallowed` (optionally `ordered`) MCP tool calls; `name` is a regex | Every capability test | -| `output-contains` / `output-not-contains` | Substring search on the final assistant message | Cheap sanity check; never the sole grader | -| `output-matches` / `output-not-matches` | Regex search on the final assistant message | Preferred output-shape grader | -| `file-exists` | A file exists in the workspace after the run | Skills that scaffold files | -| `file-contains` / `file-matches` | Substring/regex search in a workspace file | Skills that edit or generate files | -| `run-command` | Executes a shell command; nonzero exit fails the grader | Build/lint/compile verification | -| `completed` | Trajectory finished without an unhandled error | Liveness baseline | -| `prompt` | Free-form LLM judge against a custom rubric | Last resort — expensive, non-deterministic | - -Picking one: - -```text -routing? -> skill-invocation -right tool fired? -> tool-calls (required) -wrong tool did not? -> tool-calls (disallowed) -call order matters? -> tool-calls (ordered) -reply mentions X? -> output-contains (secondary only) -reply structurally right? -> output-matches -file created/edited? -> file-exists / file-contains / file-matches -build still passes? -> run-command -nothing else fits? -> prompt (last resort) -``` - -For multi-turn stimuli, a grader's `turn:` field slices the trajectory to that turn only (0-based, matching the `turns:` array index); omit it for a conversation-wide assertion. `skill-invocation`/`tool-calls` turn-scoping is reliable; `output-contains`/`output-matches` have been observed to not reliably isolate a single turn — keep those session-wide unless you've verified otherwise (per-turn grading design, tracked in microsoft/vally#481). - -## 7. Anti-patterns - -| # | Smell | Fix | -| --- | --- | --- | -| A1 | Single vacuous `output-contains` as the only grader — passes on almost any outcome | Add `skill-invocation`/`tool-calls`; keep the substring only as a secondary check | -| A2 | Negative test grades only output text, not tool absence — the tool can still fire silently | Add `tool-calls.disallowed` for every tool the skill could wrongly invoke | -| A3 | Stimulus prompt and grader substring are the same phrase — any cooperative reply parrots it | Use `output-matches` describing the response *shape*, or replace with `skill-invocation`/`tool-calls` | -| A4 | Missing/inconsistent `scoring.threshold` or `weights` not summing to `1.0` | Always set `threshold` explicitly (`0.8` convention); verify weights sum to `1.0` before adding them | -| A5 | No routing stimuli anywhere in a skill's `evals/` — routing regressions go undetected | Add ≥3 trigger + ≥3 anti-trigger stimuli (`skill-invocation` grader) to `eval.yaml` | -| A6 | Tool name typo in `required`/`disallowed` — the regex compiles, matches nothing, "passes" silently | Cross-check names against a real trajectory (`results.jsonl`) | -| A7 | Boundary anti-trigger with no competing skill mounted — trivially "passes" since nothing else was ever available | Mount every genuinely competing skill together, require the intended sibling, and disallow the skill under test | -| A8 | One environment for everything — mock can't catch contract drift against the real backend | Reserve a small live tier, run nightly, use `--workers 1` | - -## 8. Customization guidance - -- **Mock vs live**: default to mock (hermetic, PR-gate). Move to live only for behavior the mock genuinely can't represent (real auth, real backend contract) — document writes, cleanup, and run it nightly with `--workers 1`. -- **`defaults.runs`**: bump above `1` to stabilize a flaky stimulus before assuming the implementation is wrong. -- **`scoring.threshold`**: `0.8` is the starting convention for new suites; raise it as a suite's coverage and stability mature. -- **`model`**: cheaper/faster models are fine for high-volume routing suites; reserve stronger models for judgment-heavy (`prompt` grader) suites. -- **`timeout`/`--workers`**: give orchestration-heavy or live stimuli more headroom than a single tool call; `workers` is a `vally eval` CLI option, not an eval-spec field. Always serialize (`--workers 1`) anything with real side effects. - -## 9. Worked examples - -### Tool example - -```yaml -# evals/tools/prompt-to-tool-example.eval.yaml -name: azsdk-mcp-tool-invocation-eval -description: Verify prompts invoke the expected MCP tools. -type: capability -environment: azsdk-mcp-mock -defaults: - runs: 1 - timeout: "120s" - model: gpt-5.4 - executor: copilot-sdk -tags: - tier: unit - area: example -stimuli: - - name: invoke-expected-tool - prompt: "Check the status of Azure DevOps pipeline build 6447834." - graders: - - type: tool-calls - config: - required: - - name: azsdk_get_pipeline_status - disallowed: - - name: azsdk_analyze_pipeline -``` - -### Skill example - -```yaml -# .github/skills/my-skill/evals/eval.yaml -name: my-skill-eval -description: Routing and capability tests for my-skill -type: capability -environment: azsdk-mcp-mock -tags: - area: my-skill - type: ci-gate -defaults: - runs: 1 - timeout: "90s" - model: claude-opus-4.6 - executor: copilot-sdk -scoring: - threshold: 0.8 -stimuli: - # Routing - - name: trigger-obvious-phrasing - prompt: "help me do the thing my-skill exists for" - graders: - - type: skill-invocation - config: - required: ["my-skill"] - - name: anti-trigger-unrelated-request - prompt: "write a TypeSpec definition for a new resource" - graders: - - type: skill-invocation - config: - disallowed: ["my-skill"] - # Capability — same file, once routing is confirmed - - name: applies-the-expected-change - prompt: "using my-skill, do the specific thing with these concrete identifiers: ..." - graders: - - type: skill-invocation - config: - required: ["my-skill"] - - type: tool-calls - config: - required: - - name: expected_tool_call -``` - -### Workflow example - -```yaml -# evals/workflows/mock/example-multi-turn.eval.yaml -name: example-multi-turn-workflow -description: Turn 1 creates a release plan; turn 2 must reuse its result. -type: capability -environment: azsdk-mcp-mock -tags: - tier: scenario - area: release-plan -defaults: - runs: 1 - timeout: "180s" - model: gpt-5.4 - executor: copilot-sdk -stimuli: - - name: create-then-generate-conversation - environment: - skills: - - ../../../.github/skills/azsdk-common-prepare-release-plan - - ../../../.github/skills/azsdk-common-generate-sdk-locally - turns: - - prompt: "Create a public preview release plan for the Contoso Widget Manager." - - prompt: "Now generate the SDK for the plan you just created." - graders: - - type: skill-invocation - config: - required: ["azsdk-common-prepare-release-plan"] - turn: 0 - - type: skill-invocation - config: - required: ["azsdk-common-generate-sdk-locally"] - turn: 1 -``` - -## 10. Running evals locally - -Step 0's pipeline is the source of truth for this part: it tells you the repository's supported Vally installation, MCP build/setup command, `vallyRoot`, and eval glob. Do **not** copy Azure SDK Tools-specific paths into a consuming repo. - -After completing that repository's documented setup, run the focused eval from the discovered `vallyRoot`: - -```powershell -# Replace every <...> value with paths and commands from this repository's -# eval pipeline/configuration discovered in Step 0. -cd - -# Skill eval (if this repository supports skill linting): -vally lint --strict -vally eval -e /evals/eval.yaml ` - --skill-dir --workers ` - --output jsonl --output-dir - -# Tool or mock-workflow eval: -vally eval -e ` - --skill-dir --workers ` - --output jsonl --output-dir -``` - -Use `--workers 1` for live or write-side scenarios. Do not finish or open a PR until the focused local run passes. On failure, inspect `eval-results.md` and `results.jsonl` (they record every tool name/argument actually called) — fix the prompt, grader, or implementation, then rerun. Record the exact repository-specific command and pass count in the PR. diff --git a/eng/common/pipelines/skill-eval.yml b/eng/common/pipelines/skill-eval.yml index f6bdf3d9f25f..880a9d3eef22 100644 --- a/eng/common/pipelines/skill-eval.yml +++ b/eng/common/pipelines/skill-eval.yml @@ -32,10 +32,7 @@ extends: TestType: mock vallyRoot: .github/skills # Single-level glob excludes azure-typespec-author/evaluate/ (its own benchmark pipeline). - # Second entry catches the plain `eval.yaml` name (merged routing+capability file); - # it doesn't match *.eval.yaml since "eval.yaml" itself is shorter than that suffix. evalGlobs: - '*/evals/*.eval.yaml' - - '*/evals/eval.yaml' # Per-shard job timeout (report-only tier). shardTimeoutInMinutes: 20