diff --git a/skills/agent-audit/LICENSE b/skills/agent-audit/LICENSE new file mode 100644 index 00000000..4235f9ae --- /dev/null +++ b/skills/agent-audit/LICENSE @@ -0,0 +1,184 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship made available under + the License, as indicated by a copyright notice that is included in + or attached to the work (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other transformations + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean, as submitted to the Licensor for inclusion + in the Work by the copyright owner or by an individual or Legal Entity + authorized to submit on behalf of the copyright owner. For the purposes + of this definition, "submitted" means any form of electronic, verbal, + or written communication sent to the Licensor or its representatives, + including but not limited to communication on electronic mailing lists, + source code control systems, and issue tracking systems that are managed + by, or on behalf of, the Licensor for the purpose of discussing and + improving the Work, but excluding communication that is conspicuously + marked or otherwise designated in writing by the copyright owner as + "Not a Contribution." + + "Contributor" shall mean Licensor and any Legal Entity on behalf of + whom a Contribution has been received by the Licensor and subsequently + incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a cross-claim + or counterclaim in a lawsuit) alleging that the Work or any + Contribution embodied within the Work constitutes direct or contributory + patent infringement, then any patent licenses granted to You under + this License for that Work shall terminate as of the date such + litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or Derivative + Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, you must include a readable copy of the + attribution notices contained within such NOTICE file, in + at least one of the following places: within a NOTICE text + file distributed as part of the Derivative Works; within + the Source form or documentation, if provided along with the + Derivative Works; or, within a display generated by the + Derivative Works, if and wherever such third-party notices + normally appear. The contents of the NOTICE file are for + informational purposes only and do not modify the License. + You may add Your own attribution notices within Derivative + Works that You distribute, alongside or as an addendum to + the NOTICE text from the Work, provided that such additional + attribution notices cannot be construed as modifying the License. + + You may add Your own license statement for Your modifications and + may provide additional grant of rights to use, copy, modify, merge, + publish, distribute, sublicense, and/or sell copies of the + Contribution, either on an unmodified basis, with modifications, + or as part of a larger work. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or exemplary damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or all other + commercial damages or losses), even if such Contributor has been + advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + Copyright 2026 Anastasiia Stefanska + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/skills/agent-audit/SKILL.md b/skills/agent-audit/SKILL.md new file mode 100644 index 00000000..cfb284bf --- /dev/null +++ b/skills/agent-audit/SKILL.md @@ -0,0 +1,143 @@ +--- +id: agent-audit +name: agent-audit +skill-name: $agent-audit +description: Audit and proofread a Snowflake Cortex Agent's configuration against a 22-point best-practices checklist covering correctness, instruction architecture, behavioral controls, and response quality. Use when the user asks to review, audit, proofread, or improve an agent. +prompt: "$agent-audit Review my Cortex Agent for best practices" +language: en +status: Published +author: Anastasiia Stefanska +type: community +--- + +# Agent Audit + +# When to Use +- User asks to review, audit, proofread, or improve a Cortex Agent +- User wants to check their agent's configuration (instructions, orchestration, tool descriptions, tool resources, or profile) +- Do NOT use for building a new agent from scratch + +# What This Skill Provides +A comprehensive 22-point audit of a Cortex Agent's full `agent_spec`, grouped into six categories: Correctness & Hygiene, Instruction Architecture, Audience & Domain Context, Behavioral Controls, Response Quality, and Infrastructure & Robustness. Includes an optional Semantic View quality check for agents using a Cortex Analyst tool. + +# Instructions + +## Step 1: Extract the Full Agent Spec + +**Actions:** +1. **Run** `DESCRIBE AGENT ` to retrieve the full agent_spec +2. **If truncated**, extract in chunks using `SUBSTR($7, , 1000)` from `TABLE(RESULT_SCAN(LAST_QUERY_ID()))` until the full spec is captured + +**Output:** Complete agent_spec text ready for audit + +**⚠️ STOPPING POINT:** Confirm the full spec is retrieved before proceeding. + +## Step 2: Run the 22-Point Checklist + +**Actions:** +1. **Evaluate** every section of the agent_spec against all 22 checks below +2. **Record** each finding with its check number and severity + +### Correctness & Hygiene +1. **Typos & Spelling** — Scan all instruction text, profile display_name, tool names, and tool_resources for misspellings +2. **Orphan/Placeholder Text** — Look for dangling strings, leftover TODO markers, or incomplete sentences ending with "..." +3. **Valid Tool Resource References** — Verify semantic_view names, search_service names, and warehouse fields are non-empty and plausibly correct +4. **Formatting Consistency** — Check for double spaces, inconsistent newlines, mixed bullet styles, or broken markdown + +### Instruction Architecture +5. **No Cross-Layer Duplication** — Response instructions own FORMATTING; orchestration owns ROUTING/LOGIC; tool descriptions own CAPABILITY SCOPE. Flag any rule that appears in more than one layer. +6. **Guardrails Positioned First** — Off-topic refusal, scope boundaries, and safety rules must appear at the TOP of orchestration instructions, not buried in the middle or end +7. **Consolidated "Do Not" Rules** — All negative constraints must live in ONE authoritative list, not scattered across layers + +### Audience & Domain Context +8. **Audience Definition Present** — Orchestration must explicitly name primary users and state what they value most +9. **Domain Context Present** — Instructions must name the specific business domain, data domains, or organization +10. **Numbers-in-Context Rule** — Every metric must include denominator/sample context (e.g., "67% based on 2 wins out of 3 closed deals," not just "67%") + +### Behavioral Controls +11. **Result Count Cap** — Instructions must explicitly limit how many results are displayed (e.g., "Return at most 5 results unless the user requests more") +12. **Explicit Tool Priority** — Orchestration must state tool selection order clearly (e.g., "Always use Analyst first for quantitative questions") +13. **Keyword Signal Lists** — Orchestration must list trigger words mapped to each tool (quantitative signals → Analyst; qualitative signals → Search) +14. **Sequential Multi-Tool Pattern** — For questions filtering unstructured data by structured criteria, orchestration must enforce query-then-search: first query Analyst for record identifiers, then pass those into the Search query +15. **Empty-Results Handling** — If a structured query returns no data, the agent must report that gap honestly and NOT silently fall back to search +16. **Graceful Edge Cases** — Explicit handling must exist for: no results found, ambiguous queries, and multiple possible interpretations + +### Response Quality +17. **Response Format Guidance** — Response instructions must specify structure: lead with a direct answer first, then supporting details; prose over bullet points +18. **Concrete Citation Format** — Require specific citations ("In the TechCorp discovery call"), not vague ones ("according to the data") +19. **Number Formatting Rules** — Standardize number display (e.g., deal values in K suffix: $90,000 → 90K; percentages rounded to whole numbers) +20. **Data Limitation Transparency** — When synthesizing fewer than three data points, the agent must note this explicitly (e.g., "based on two closed Enterprise deals") + +### Infrastructure & Robustness +21. **Pinned Orchestration Model** — `models.orchestration` must be a specific model name, not `"auto"`. Flag `"auto"` as a risk. +22. **Tuned max_results** — Values below 5 may be too restrictive; values above 10 may flood context. Recommend 5–6 for most catalog use cases. + +**Output:** Full list of findings with check number and severity + +**⚠️ STOPPING POINT:** Present findings to the user before suggesting any fixes. + +## Step 3: Report Findings by Severity + +**Actions:** +1. **Group** all findings into three severity tiers and present as a table: + - **Critical** — Will cause incorrect behavior, silent failures, or broken tool calls (e.g., typo in tool_resource name, missing tool priority, no guardrails) + - **High** — Will cause poor user experience or unreliable responses (e.g., no audience definition, missing citation format, no sequential multi-tool pattern) + - **Nice-to-have** — Polish and robustness improvements (e.g., keyword signal lists, number formatting rules, result count cap) +2. **Confirm** if all 22 checks pass: output "All 22 best-practice checks passed." + +**Output:** Severity-grouped findings table + +## Step 4: Semantic View Quality Checks (if applicable) + +**Actions:** If the agent uses a Cortex Analyst tool backed by a semantic view, also: +1. **Check** synonyms are defined on all key dimensions (customer name, sales stage, sales rep, product line) +2. **Check** custom measures are present for any KPI requiring a formula (e.g., win rate = SUM(CASE WHEN win_status = true THEN 1 ELSE 0 END) / COUNT(*)) +3. **Check** sample values are enabled on the semantic view so Cortex Analyst can infer data ranges +4. **Check** AI-generated descriptions are enabled on tables and columns to reduce query planning ambiguity + +**Output:** Semantic view findings appended to the main report + +## Best Practices +- Always extract the FULL agent_spec before auditing — truncated specs will miss issues +- Pay special attention to tool_resources: typos there cause silent failures +- Check the profile section (display_name, comment) as part of the audit +- Non-determinism is normal — if a use case requires consistent behavior, recommend more detailed orchestration instructions with explicit step-by-step patterns + +## Instruction Layer Quick-Reference + +Use this table to decide where a rule belongs before auditing or writing instructions. + +| Layer | Owns | Does NOT own | +|---|---|---| +| **Orchestration instructions** | Tool selection logic, keyword signals, sequential patterns, scope boundaries, guardrails, audience definition | Output formatting, citation style, number formatting | +| **Response instructions** | Output format, tone, citation style, number formatting, data limitation transparency | Routing logic, tool selection, guardrails | +| **Tool descriptions** | What data the tool contains, when it is appropriate to use | Formatting rules, routing hierarchies that span multiple tools | + +## Iteration Loop (Trace-Driven Improvement) + +After testing the agent, use this loop to fix issues systematically: + +1. **Identify symptom** — What did the agent do wrong? (wrong tool, bad format, speculation, vague citation) +2. **Diagnose the layer** — Wrong tool selection → fix orchestration. Poor response format → fix response instructions. Scope violation → fix boundary rules. +3. **Write a specific fix** — Include a pattern, a rationale, and an example; vague instructions produce inconsistent behavior +4. **Retest via trace** — Confirm the planning span reflects the intended tool call sequence and the response span reflects the format rules +5. **Document the pattern** — Keep a log of issues and fixes; over time this becomes a reusable instruction library + +# Stopping Points +- ✋ After Step 1 — confirm full spec is retrieved before running checks +- ✋ After Step 2 — present all findings to the user before suggesting fixes + +**Resume rule:** Upon user approval, proceed directly to the next step without re-asking. + +# Output +A severity-grouped audit report (Critical / High / Nice-to-have) covering all 22 checks, plus optional Semantic View quality findings. If all checks pass, outputs: "All 22 best-practice checks passed." + +# Examples + +## Example 1: Basic audit +User: `$agent-audit Review my Cortex Agent for best practices` +Assistant: Prompts for the agent name, runs `DESCRIBE AGENT `, extracts the full spec, evaluates all 22 checks, and returns a grouped findings table. + +## Example 2: Audit with a Cortex Analyst tool +User: `$agent-audit Audit my sales agent including the semantic view` +Assistant: Runs all 22 checks plus the four Semantic View Quality Checks, and reports all findings grouped by severity.