Sync .github/skills directory with azure-sdk-tools for PR 16412

.github/skills/azsdk-common-eval-authoring-skill/SKILL.md

@@ -0,0 +1,40 @@
+---
+name: azsdk-common-eval-authoring-skill
+description: 'Author and validate Vally evals for Agent Skills under .github/skills. WHEN: "write a skill eval", "add trigger eval", "test skill routing", "add anti-trigger tests", "create skill capability eval", "harden skill graders". DO NOT USE FOR: single MCP tool prompt-to-tool coverage (use azsdk-common-eval-authoring-tool), multi-tool or multi-turn scenarios (use azsdk-common-eval-authoring-workflow), authoring SKILL.md itself (use skill-authoring).'
+license: MIT
+metadata:
+  author: Microsoft
+  version: "1.0.0"
+  distribution: shared
+compatibility: "copilot-chat, @microsoft/vally-cli 0.7.0"
+---
+
+# Skill Eval Authoring
+
+Author Vally evals that verify **one Agent Skill's** routing and behavior in isolation, under `.github/skills/<skill>/evals/`. All the actual guidance — naming convention, placement, per-category requirements, glossary, grading patterns, grader catalog, anti-patterns, and worked examples — lives in one shared place so it never drifts across the three eval-authoring skills: the eval authoring guide at `eng/common/knowledge/eval-authoring/README.md`. This skill exists to route you there with skill-eval context already loaded, not to duplicate it.
+
+## Triggers
+
+USE FOR: write a skill eval, add trigger eval, test skill routing, add anti-trigger tests, create skill capability eval, harden skill graders
+WHEN: "write a skill eval", "add trigger eval", "test skill routing", "add anti-trigger tests", "create skill capability eval", "harden skill graders"
+DO NOT USE FOR: single MCP tool prompt-to-tool coverage (use azsdk-common-eval-authoring-tool), multi-tool or multi-turn scenarios (use azsdk-common-eval-authoring-workflow), authoring SKILL.md itself (use skill-authoring)
+
+## Steps
+
+1. Read the eval authoring guide (`eng/common/knowledge/eval-authoring/README.md`) Step 0 to find this repo's `vallyRoot`/`evalGlobs` for the skill tier, then the guide's "Skill" column throughout (naming, requirements, worked example).
+2. Read the target `SKILL.md` — its `WHEN`/`DO NOT USE FOR` boundaries, invoked tools — and its existing `evals/`.
+3. Write `eval.yaml` with routing stimuli (trigger + anti-trigger, `skill-invocation` grader) and capability stimuli together in the same file per the guide's naming convention and four-layer pattern. Split into additional `<behavior>.eval.yaml` files only once a skill's coverage genuinely grows large. For a boundary prompt, mount and **require** the intended competing skill while disallowing the skill under test — an anti-trigger with no competing skill mounted trivially "passes" (guide anti-pattern A7).
+4. Supply concrete identifiers (repo path, package name, anything needed to reach the call) in every stimulus prompt — a vague prompt fails `tool-calls` with zero recorded calls, which looks like a routing bug but is really a missing-context prompt bug.
+5. Validate locally per the guide's "Running evals locally" section. Do not finish or open a PR until the focused local eval passes; inspect `eval-results.md` and `results.jsonl` on failure.
+
+## Rules
+
+- Cover both activation and non-activation; anti-triggers should target neighboring skills, not random unrelated prompts.
+- Omit `turn` for conversation-wide assertions; pin a turn only when that turn owns the behavior unambiguously.
+- Never copy secrets or production-write identifiers into fixtures.
+- Update existing specs additively unless the requested contract changed.
+
+## References
+
+- Eval authoring guide: `eng/common/knowledge/eval-authoring/README.md` — naming, placement, requirements, glossary, graders, anti-patterns, worked examples, local commands. Shared with `azsdk-common-eval-authoring-tool`/`azsdk-common-eval-authoring-workflow` — update it there, not per-skill.
+- Repository-local eval README/configuration discovered in Step 0 (if present)

.github/skills/azsdk-common-eval-authoring-skill/evals/eval.yaml

@@ -0,0 +1,123 @@
+name: azsdk-common-eval-authoring-skill-eval
+description: Routing and boundary tests for the azsdk-common-eval-authoring-skill skill
+type: capability
+
+environment: azsdk-mcp-mock
+
+tags:
+  area: azsdk-common-eval-authoring-skill
+  type: ci-gate
+
+defaults:
+  runs: 1
+  timeout: "120s"
+  model: claude-opus-4.6
+  executor: copilot-sdk
+
+scoring:
+  threshold: 0.8
+
+stimuli:
+  # Boundary anti-triggers need the two sibling eval-authoring skills mounted
+  # alongside this one, or "disallowed" is trivially satisfied because no
+  # better-fitting skill was ever available to the model.
+  - name: trigger-write-skill-eval
+    environment: &siblingSkills
+      skills:
+        - ..
+        - ../../azsdk-common-eval-authoring-tool
+        - ../../azsdk-common-eval-authoring-workflow
+    prompt: "write a robust eval for my Agent Skill"
+    graders:
+      - type: skill-invocation
+        config:
+          required: [azsdk-common-eval-authoring-skill]
+          disallowed:
+            [
+              azsdk-common-eval-authoring-tool,
+              azsdk-common-eval-authoring-workflow,
+            ]
+
+  - name: trigger-add-trigger-and-anti-trigger-tests
+    environment: *siblingSkills
+    prompt: "Add Vally trigger and anti-trigger routing tests for .github/skills/azsdk-common-generate-sdk-locally. Its eval file is .github/skills/azsdk-common-generate-sdk-locally/evals/eval.yaml and it handles local TypeSpec SDK generation."
+    graders:
+      - type: skill-invocation
+        config:
+          required: [azsdk-common-eval-authoring-skill]
+          disallowed:
+            [
+              azsdk-common-eval-authoring-tool,
+              azsdk-common-eval-authoring-workflow,
+            ]
+
+  - name: trigger-test-skill-routing
+    environment: *siblingSkills
+    prompt: "Test routing for .github/skills/azsdk-common-pipeline-analysis: it should activate for Azure SDK CI/pipeline diagnosis requests but not for requests to fix the failure. Add the appropriate Vally routing eval cases."
+    graders:
+      - type: skill-invocation
+        config:
+          required: [azsdk-common-eval-authoring-skill]
+          disallowed:
+            [
+              azsdk-common-eval-authoring-tool,
+              azsdk-common-eval-authoring-workflow,
+            ]
+
+  - name: trigger-harden-skill-graders
+    environment: *siblingSkills
+    prompt: "harden my skill capability eval so refusals cannot pass"
+    graders:
+      - type: skill-invocation
+        config:
+          required: [azsdk-common-eval-authoring-skill]
+          disallowed:
+            [
+              azsdk-common-eval-authoring-tool,
+              azsdk-common-eval-authoring-workflow,
+            ]
+
+  - name: anti-trigger-single-mcp-tool-eval
+    environment: *siblingSkills
+    prompt: "add a prompt-to-tool eval for azsdk_get_release_plan under evals/tools"
+    graders:
+      - type: skill-invocation
+        config:
+          required: [azsdk-common-eval-authoring-tool]
+          disallowed:
+            [
+              azsdk-common-eval-authoring-skill,
+              azsdk-common-eval-authoring-workflow,
+            ]
+
+  - name: anti-trigger-multi-tool-workflow
+    environment: *siblingSkills
+    prompt: "write a mock workflow eval that checks two MCP tools are called in order"
+    graders:
+      - type: skill-invocation
+        config:
+          required: [azsdk-common-eval-authoring-workflow]
+          disallowed:
+            [
+              azsdk-common-eval-authoring-skill,
+              azsdk-common-eval-authoring-tool,
+            ]
+
+  - name: anti-trigger-author-skill
+    environment:
+      skills:
+        - ..
+        - ../../azsdk-common-eval-authoring-tool
+        - ../../azsdk-common-eval-authoring-workflow
+        - ../../skill-authoring
+    prompt: "create a new Agent Skill and write its SKILL.md"
+    graders:
+      - type: skill-invocation
+        config:
+          required: [skill-authoring]
+          disallowed:
+            [
+              azsdk-common-eval-authoring-skill,
+              azsdk-common-eval-authoring-tool,
+              azsdk-common-eval-authoring-workflow,
+            ]

.github/skills/azsdk-common-eval-authoring-tool/SKILL.md

@@ -0,0 +1,41 @@
+---
+name: azsdk-common-eval-authoring-tool
+description: 'Author and validate hermetic single-tool Vally evals under evals/tools. WHEN: "write a tool eval", "add prompt-to-tool coverage", "test MCP tool selection", "add tool catalog eval", "create single-tool scenario", "harden tool-call grader". DO NOT USE FOR: skill routing or capability evals (use azsdk-common-eval-authoring-skill), multi-tool, multi-turn, or live scenarios (use azsdk-common-eval-authoring-workflow).'
+license: MIT
+metadata:
+  author: Microsoft
+  version: "1.0.0"
+  distribution: shared
+compatibility: "copilot-chat, @microsoft/vally-cli 0.7.0"
+---
+
+# Tool Eval Authoring
+
+Author Vally evals that verify **one MCP tool** is selected correctly for a given prompt, under `evals/tools/`. All the actual guidance — naming convention, placement, per-category requirements, glossary, grading patterns, grader catalog, anti-patterns, and worked examples — lives in one shared place so it never drifts across the three eval-authoring skills: the eval authoring guide at `eng/common/knowledge/eval-authoring/README.md`. This skill exists to route you there with tool-eval context already loaded, not to duplicate it.
+
+## Triggers
+
+USE FOR: write a tool eval, add prompt-to-tool coverage, test MCP tool selection, add tool catalog eval, create single-tool scenario, harden tool-call grader
+WHEN: "write a tool eval", "add prompt-to-tool coverage", "test MCP tool selection", "add tool catalog eval", "create single-tool scenario", "harden tool-call grader"
+DO NOT USE FOR: skill routing or capability evals (use azsdk-common-eval-authoring-skill), multi-tool, multi-turn, or live scenarios (use azsdk-common-eval-authoring-workflow)
+
+## Steps
+
+1. Read the eval authoring guide (`eng/common/knowledge/eval-authoring/README.md`) Step 0 to find this repo's `vallyRoot`/`evalGlobs` for the tool tier, then the guide's "Tool" column throughout (naming, requirements, worked example).
+2. Confirm the scenario expects one primary tool. If success requires orchestration, conversation state, or live services, use `azsdk-common-eval-authoring-workflow` instead.
+3. Add stimuli to the matching `prompt-to-tool-<area>.eval.yaml`; create a separate file only when it needs fixtures or outcome grading.
+4. Use realistic, concrete prompts with multiple natural phrasings and collision cases that disallow the nearest competing tool. Make `tool-calls` the primary signal; avoid the anti-patterns in the guide (vacuous keyword-only grading, missing `scoring.threshold`).
+5. Keep the unit tier hermetic (`environment: azsdk-mcp-mock`, `tags.tier: unit`, established area tag). Avoid git worktrees and production writes.
+6. Validate locally per the guide's "Running evals locally" section, building the mock MCP first. Do not finish or open a PR until it passes; inspect recorded tool names and arguments on failure.
+
+## Rules
+
+- Use exact MCP tool names and assert forbidden alternatives where ambiguity exists.
+- Do not force a tool through unnatural prompt instructions unless direct invocation is the contract being tested.
+- Keep fixture paths relative to the eval file and fixture data minimal and non-secret.
+- Preserve existing namespace coverage and add new stimuli instead of duplicating files.
+
+## References
+
+- Eval authoring guide: `eng/common/knowledge/eval-authoring/README.md` — naming, placement, requirements, glossary, graders, anti-patterns, worked examples, local commands. Shared with `azsdk-common-eval-authoring-skill`/`azsdk-common-eval-authoring-workflow` — update it there, not per-skill.
+- Repository-local eval README/configuration discovered in Step 0 (if present)

.github/skills/azsdk-common-eval-authoring-tool/evals/eval.yaml

@@ -0,0 +1,117 @@
+name: azsdk-common-eval-authoring-tool-eval
+description: Routing and boundary tests for the azsdk-common-eval-authoring-tool skill
+type: capability
+
+environment: azsdk-mcp-mock
+
+tags:
+  area: azsdk-common-eval-authoring-tool
+  type: ci-gate
+
+defaults:
+  runs: 1
+  timeout: "120s"
+  model: claude-opus-4.6
+  executor: copilot-sdk
+
+scoring:
+  threshold: 0.8
+
+stimuli:
+  # Boundary anti-triggers need the two sibling eval-authoring skills mounted
+  # alongside this one, or "disallowed" is trivially satisfied because no
+  # better-fitting skill was ever available to the model.
+  - name: trigger-write-tool-eval
+    environment: &siblingSkills
+      skills:
+        - ..
+        - ../../azsdk-common-eval-authoring-skill
+        - ../../azsdk-common-eval-authoring-workflow
+    prompt: "write a Vally eval for a single Azure SDK MCP tool"
+    graders:
+      - type: skill-invocation
+        config:
+          required: [azsdk-common-eval-authoring-tool]
+          disallowed:
+            [
+              azsdk-common-eval-authoring-skill,
+              azsdk-common-eval-authoring-workflow,
+            ]
+
+  - name: trigger-add-prompt-to-tool-coverage
+    environment: *siblingSkills
+    prompt: "add prompt-to-tool coverage for azsdk_get_release_plan"
+    graders:
+      - type: skill-invocation
+        config:
+          required: [azsdk-common-eval-authoring-tool]
+          disallowed:
+            [
+              azsdk-common-eval-authoring-skill,
+              azsdk-common-eval-authoring-workflow,
+            ]
+
+  - name: trigger-test-mcp-tool-selection
+    environment: *siblingSkills
+    prompt: "test whether user prompts select the correct MCP tool"
+    graders:
+      - type: skill-invocation
+        config:
+          required: [azsdk-common-eval-authoring-tool]
+          disallowed:
+            [
+              azsdk-common-eval-authoring-skill,
+              azsdk-common-eval-authoring-workflow,
+            ]
+
+  - name: trigger-harden-tool-call-grader
+    environment: *siblingSkills
+    prompt: "harden this single-tool eval against a competing tool call"
+    graders:
+      - type: skill-invocation
+        config:
+          required: [azsdk-common-eval-authoring-tool]
+          disallowed:
+            [
+              azsdk-common-eval-authoring-skill,
+              azsdk-common-eval-authoring-workflow,
+            ]
+
+  - name: anti-trigger-skill-routing
+    environment: *siblingSkills
+    prompt: "add trigger and anti-trigger tests to verify my Agent Skill activates for the right prompts"
+    graders:
+      - type: skill-invocation
+        config:
+          required: [azsdk-common-eval-authoring-skill]
+          disallowed:
+            [
+              azsdk-common-eval-authoring-tool,
+              azsdk-common-eval-authoring-workflow,
+            ]
+
+  - name: anti-trigger-multi-tool-workflow
+    environment: *siblingSkills
+    prompt: "create an end-to-end eval for a sequence of three MCP tools"
+    graders:
+      - type: skill-invocation
+        config:
+          required: [azsdk-common-eval-authoring-workflow]
+          disallowed:
+            [
+              azsdk-common-eval-authoring-skill,
+              azsdk-common-eval-authoring-tool,
+            ]
+
+  - name: anti-trigger-live-scenario
+    environment: *siblingSkills
+    prompt: "add a nightly live eval that writes to Azure DevOps"
+    graders:
+      - type: skill-invocation
+        config:
+          required: [azsdk-common-eval-authoring-workflow]
+          disallowed:
+            [
+              azsdk-common-eval-authoring-skill,
+              azsdk-common-eval-authoring-tool,
+            ]

.github/skills/azsdk-common-eval-authoring-workflow/SKILL.md

@@ -0,0 +1,41 @@
+---
+name: azsdk-common-eval-authoring-workflow
+description: 'Author and validate multi-tool, multi-turn, mock, and live Vally scenarios under evals/workflows. WHEN: "write a workflow eval", "add end-to-end eval", "test tool sequence", "create multi-turn eval", "add mock workflow scenario", "add live eval". DO NOT USE FOR: per-skill routing/capability evals (use azsdk-common-eval-authoring-skill), isolated prompt-to-tool tests (use azsdk-common-eval-authoring-tool).'
+license: MIT
+metadata:
+  author: Microsoft
+  version: "1.0.0"
+  distribution: shared
+compatibility: "copilot-chat, @microsoft/vally-cli 0.7.0"
+---
+
+# Workflow Eval Authoring
+
+Author Vally evals that verify **multi-skill, multi-tool, or multi-turn** orchestration, state, and outcomes across an entire agent conversation, under `evals/workflows/`. All the actual guidance — naming convention, placement, per-category requirements, glossary, grading patterns, grader catalog, anti-patterns, and worked examples — lives in one shared place so it never drifts across the three eval-authoring skills: the eval authoring guide at `eng/common/knowledge/eval-authoring/README.md`. This skill exists to route you there with workflow-eval context already loaded, not to duplicate it.
+
+## Triggers
+
+USE FOR: write a workflow eval, add end-to-end eval, test tool sequence, create multi-turn eval, add mock workflow scenario, add live eval
+WHEN: "write a workflow eval", "add end-to-end eval", "test tool sequence", "create multi-turn eval", "add mock workflow scenario", "add live eval"
+DO NOT USE FOR: per-skill routing/capability evals (use azsdk-common-eval-authoring-skill), isolated prompt-to-tool tests (use azsdk-common-eval-authoring-tool)
+
+## Steps
+
+1. Read the eval authoring guide (`eng/common/knowledge/eval-authoring/README.md`) Step 0 to find this repo's `vallyRoot`/`evalGlobs` for the workflow tier, then the guide's "Workflow" column throughout (naming, requirements, worked example).
+2. Use `evals/workflows/mock/` by default. Choose `live/` only when the behavior cannot be represented by the mock MCP; document writes, authentication, cleanup, and nightly-only execution.
+3. Model one user goal per stimulus. Use `turns` only when conversation state matters; otherwise keep a single prompt. Mount every candidate skill explicitly, provide minimal file/git fixtures, and bound turns, tokens, workers, and timeout.
+4. Combine process and outcome graders per the guide's four-layer pattern and grader catalog: required/disallowed skills and tools, ordering where essential, files/commands, response quality. Avoid overfitting to incidental call sequences and the guide's anti-patterns (e.g. boundary anti-triggers with no competing skill mounted).
+5. Scope graders to `turn` only when that turn owns the assertion unambiguously; otherwise grade the full conversation.
+6. Validate locally per the guide's "Running evals locally" section, building the matching MCP and priming git fixtures when required. Do not finish or open a PR until it passes; inspect every turn and tool call on failure.
+
+## Rules
+
+- Keep mock workflows hermetic and repeatable; use fake IDs and canned responses.
+- Never run a live write scenario without the documented test area and safety environment.
+- Preserve relative path invariants for `environment.skills`, `files`, and `git.source`.
+- Use weights summing to `1.0` and a meaningful threshold when combining grader types.
+
+## References
+
+- Eval authoring guide: `eng/common/knowledge/eval-authoring/README.md` — naming, placement, requirements, glossary, graders, anti-patterns, worked examples, local commands. Shared with `azsdk-common-eval-authoring-skill`/`azsdk-common-eval-authoring-tool` — update it there, not per-skill.
+- Repository-local eval README/configuration discovered in Step 0 (if present)