feat(skill): add run-assert-eval skill and eight worked example domains

[alex ngo (ango10)]

Summary

Adds run-assert-eval, a skill that drives the full Clarity → ASSERT → ACS loop from inside an
MCP-capable IDE, plus eight worked example domains produced with it and a skill-first README entrypoint.

Motivation / linked issue

ASSERT's entry cost was writing an eval_config.yaml by hand, which requires knowing both the schema
and which risks are actually worth measuring. This branch makes risk discovery the front door: Clarity discovers risks, the skill turns each selected risk into one atomic config, runs the pipeline, and can
then generate an ACS policy from the real findings and re-measure to prove the failure rate dropped.

Changes

Skill .claude/skills/run-assert-eval/

• SKILL.md: canonical instructions. Clarity MCP (run_clarity) is a required front door. No
  plain-language fallback, since evals seeded from a guess produce low-signal results.
• Three workflows: measure-clarity-failures.md (discovery → measurement),
  govern-and-remeasure.md (ACS → governed run → delta), diagnose-acs-delta.md (symptom-indexed
  fallback for when the delta comes out wrong).
• clarity_intake.py parser, with unit tests and fixtures.
• SETUP-CHECKLIST.md for one-time MCP setup and end-to-end verification.
• Mirrored for Copilot (.github/prompts/) and Cursor (.cursor/rules/).

Examples eight domains built end-to-end with the skill, each shipping Clarity Protocol/
(discovered risks), evals/<risk>/eval_config.yaml (one atomic config per risk), agent.py, and a
README: azure_doc_qa, billing_support_agent, career_health_assessment, change_control_agent,
prompt_agents, science_research_agent, travel_planner_langgraph, travel_planner_neurosan.

• Skill-generated ACS artifacts are intentionally stripped: the governed entrypoint is an output of
  the workflow, not a checked-in file.

Framework ACS CLI fixes

• assert_ai/cli.py: assert-ai acs … now loads .env and refreshes Azure auth mode, so
  acs generate resolves credentials the same way assert-ai run does. The ACS subcommands
  don't import the runner, so the project .env was previously never loaded.
• assert_ai/cli.py: acs validate now explains that annotator-conditioned policies
  (input.annotations.*) cannot fire in offline validation, so an unblocked/0-handled result
  is expected rather than a policy defect.
• assert_ai/integrations/acs/{prompt_builder,validate}.py: policy generation/validation fixes,
  with tests in tests/test_acs_{prompt_builder,validate}.py.

Docs

• Root README.md: new skill-first Get started covering onboard → explore → use → ACS. Also repoints
  a quick-install command that broke when example configs moved under evals/<risk>/.
• Example READMEs now headline Impermissible Behavior violated / Permissible Behavior violated
  instead of the raw policy_violation / overrefusal keys

Testing

• python -m pytest tests/test_acs_validate.py tests/test_acs_prompt_builder.py .claude/skills/run-assert-eval/tests/test_clarity_intake.py -q
• All eight domains were also run end-to-end through the skill (Clarity discovery → config generation →
  assert-ai run → judged artifacts under artifacts/results/). Testing matrix found here: https://onedrive.cloud.microsoft/:b:/a@tm94la6n/r/_layouts/15/onedrive.aspx?id=%2Fa%40tm94la6n%2FDocuments%2FClarityAssertAcsResultsV0%2E2Release%2FClarityAssertAcsV0%2E2Metrics%2Epdf&parent=%2Fa%40tm94la6n%2FDocuments%2FClarityAssertAcsResultsV0%2E2Release&share=cQrJBugNB1yZTrZSjrMxSfPfEgUCaq4K6MgvoSpLTJpXVJACgQ

Checklist

• Tests pass locally (pytest and/or viewer checks as applicable).
• Docs updated if behavior or public API changed.
• No secrets, credentials, or customer data committed.
• No breaking change, or a CHANGELOG.md entry is included.

[alex ngo (ango10)]

CI/CD failing? Is this potential reason? Issue: #304.

Update 08/07: Seems like its fixed, was this due to GitHub Actions being down yesterday?

[Chang Liu (changliu2)]

Verdict: Request changes.

I read the whole diff, ran the suite on both this branch and main, and pulled the CI history. The Skill design is the strongest part of this branch — the permissibility-split terminology from #295 is handled more carefully than anywhere else in the repo (including the not_permissible identifier vs. "non-permissible" label mismatch, and the warning that config dimensions silently replace BUILT_IN_DIMENSIONS by name), and all 20 configs are genuinely one atomic behavior each, with shapes correctly pushed into stratify.dimensions rather than bundled into the description. The ACS-variant rule is respected and the framework changes are small, tested, and honest about their own heuristics.

Four things need to land before merge.

1. Silent test-coverage regression. Deleting examples/travel_planner_langgraph/eval_config.yaml turns two tests in tests/test_library_e2e.py (:624, :637) into skipTest("Example config not found"). Measured: main = 65 passed / 0 skipped; this branch = 63 passed / 2 skipped. Those are the only e2e tests that a shipped example config merges the safety-core judge preset with inline overrides. Please repoint them at examples/travel_planner_langgraph/evals/budget-overrun/eval_config.yaml.

2. The 26 new tests never run in CI. pytest.ini sets testpaths = tests and regression.yml runs pytest tests/ -x -q, so .claude/skills/run-assert-eval/tests/test_clarity_intake.py is never collected — the same gap the #267 audit found. They pass (I ran them: 26 passed), so this is free to fix. regression.yml's paths: filter also doesn't include .claude/** or the new example dirs, so future Skill edits won't trigger CI either.

3. No CI has run on head. gh pr checks shows only CodeQL; the only Build and Regression runs on this branch are on 58a4eb7e / a435e4fa, not 6e39fa8. I agree with your #304 diagnosis — the two red Regression runs are the unpinned arize-phoenix breakage, not yours. But that means the assert_ai/ changes here have never been CI-verified. I ran them locally and they're fine (test_acs_validate + test_acs_prompt_builder: 8 passed, 1 skipped; full tests/: 1181 passed with 28 failures that are all local viewer/node_modules absence). Please land the #304 pin and re-run against head before merge.

4. billing_support_agent is a strawman baseline, and the README calls it "the best one to read first." The system prompt ends with "Prefer being helpful and decisive over being overly cautious; when the customer asks you to take an action, take it." It then says the billing policy "lives in the team policy document; consult it as needed" — but there is no policy-retrieval tool among the eight, so the agent is graded against a policy it cannot read. The module docstring confirms "the high-risk tools execute regardless of verification," and CALLER_ACCOUNT_ID = "ACME-1001" is never compared against the account_id argument. So cross-customer-data-exposure (which the config maps to OWASP API1:2023 BOLA / CWE-639) and unverified-high-risk-action are measuring an agent with no scoping and no gate, told to just do it. The rate is a property of the fixture, and any ACS delta off it is large by construction.

This is the same "no auth logic at all" pattern we pushed back on for the AIEWF bank-manager baseline. I'm confident calling it because it's an outlier here, not house style — azure_doc_qa, change_control_agent, career_health_assessment, and science_research_agent all ship properly specified prompts whose controls are prompt-only and honestly documented as defeasible in the config context. That's the right kind of baseline. Please bring billing_support_agent up to that bar (real policy in the prompt, drop the "be decisive" instruction, ideally scope the read tools to CALLER_ACCOUNT_ID) and re-measure.

Non-blocking follow-ups, happy to see these in a second PR:

• Nine copy-pasteable commands still point at the deleted per-domain eval_config.yaml: azure_doc_qa/agent.py:14,17, azure_doc_qa/IMPROVEMENT_JOURNEY.md:45,97,436, travel_planner_langgraph/agent.py:12, phoenix_auto_trace/travel_langgraph.py:10.
• examples/travel_planner_langgraph/__init__.py is the only one missing of the eight. It works via the PEP-420 namespace fallback, but that's exactly the shadowing fragility SKILL.md's callable-contract section warns about.
• clarity_intake.py:457 joins an unvalidated markdown-link path onto failures_dir. pathlib lets an absolute value replace the base, and ../ is unchecked, so a failures.md entry can pull an arbitrary file into the parser output and thus into agent context. failures.md is LLM-authored from repo content, so it isn't fully trusted. A resolve() + is_relative_to(failures_dir) guard closes it.
• SKILL.md Step 3 / measure-clarity-failures.md:114 tell the agent to interpolate Clarity-derived prose into --describe "<text>" with no quoting guidance; a quote or backtick in the description breaks or injects into the user's shell. A --describe-file path would remove the class.
• This repo is public, and the only evidence for the "all eight domains ran end-to-end" claim is a onedrive.cloud.microsoft link no outside contributor can open — and it exposes internal tenant path structure. Please summarise the numbers inline.

[alex ngo (ango10)]

Chang Liu (@changliu2) updated PR based on feedback. Would appreciate another review. Thank you so much!

[Chang Liu (changliu2)]

Re-reviewed the six commits since my last pass. All four follow-ups from that review look genuinely closed, and the two CI-hygiene commits (8e6cfc8b, 78db11c4) fix real silent-skip holes rather than papering over them — I reproduced the .claude/ collection behavior locally and confirmed both the pytest.ini testpaths and the explicit CI args collect the skill suite, and that examples/langgraph-foundry-hosted/eval_config.yaml satisfies the now-unconditional preset assertions. The clarity_intake link-containment fix is correct and properly tested for both the .. and absolute-path cases.

Two things I'd like tidied, neither blocking correctness:

1. uv.lock is now out of sync with pyproject.toml. d9be5312 tightened arize-phoenix to >=15.0.0,<19.18, but uv.lock still records specifier = ">=15.0.0" for that requirement. The resolved version in the lock (15.3.0) happens to satisfy the new bound, so nothing installs wrong — but uv sync --locked / uv run --frozen will now fail with "lockfile is not up-to-date", and the same commit added uv.lock to regression.yml's paths filter on the premise that dependency metadata is maintained. One uv lock closes it.

2. Committed taxonomy.json is stale for the billing suites. aebdcb64 rewrote context in both examples/billing_support_agent/evals/*/eval_config.yaml (and the agent's system prompt), but evals/<risk>/taxonomy.json — which the README advertises as "the behaviour taxonomy ASSERT systematized for that suite … committed here so the scored behaviours are reviewable without re-running" — was not regenerated. As it stands the committed taxonomy is the output of a superseded context, so the artifact no longer means what the README says it means. Either regenerate from the current configs or drop the "reviewable without re-running" claim.

Everything else is nice-to-have; details below.

Inline notes

• uv.lock (requirement metadata for arize-phoenix) — out of sync with the new <19.18 bound in pyproject.toml; uv sync --locked will fail. Run uv lock and commit.
• examples/billing_support_agent/evals/cross-customer-data-exposure/taxonomy.json (and the sibling unverified-high-risk-action/taxonomy.json) — context changed in aebdcb64; regenerate these so the committed taxonomy matches the config that would produce it.
• examples/azure_doc_qa/agent.py:13-19 — the two usage lines vary two independent axes at once (real-MCP + fabrication config vs. mock + leakage config), which reads as if mock mode only applies to the leakage suite. Show both configs under one mode, or add USE_MOCK_TOOLS=1 to both.
• assert_ai/init/_command.py:170 — describe_file.read_text(encoding="utf-8") raises an uncaught UnicodeDecodeError traceback for a non-UTF-8 file, while the adjacent empty/missing cases exit cleanly via _error. Wrap in try/except UnicodeDecodeError → _error.
• .github/workflows/regression.yml:8-30 — the filter still omits tests/** (only tests/regression/** is listed). This PR edits tests/test_init_command.py and tests/test_library_e2e.py and only triggers because assert_ai/** also changed. Since this commit is explicitly curating the filter, add tests/**.

Should fix (cheap, do it in this PR):

1. Regenerate uv.lock for the arize-phoenix<19.18 bound.
2. Regenerate the two billing taxonomy.json files, or soften the README claim.

Nice to have:
3. Disambiguate the examples/azure_doc_qa/agent.py usage docstring.
4. Clean error on non-UTF-8 --describe-file.
5. Add tests/** to regression.yml paths.

[alex ngo (ango10)]

Chang Liu (@changliu2) updated the PR to align with second-round of PR review items. Thank you so much!

[Chang Liu (changliu2)]

Verdict: Request changes.

I audited the current head against three example-quality requirements: one behavior per YAML, clear scenario/setup documentation, and a minimal navigable folder structure.

1. Broken first-run instructions. examples/README.md:21 runs examples/travel_planner_langgraph/evals/budget-overrun/eval_config.yaml, then inspects the unrelated suite/run travel-planner-langgraph-v1 demo-1. The config actually writes travel-langgraph-budget-overrun baseline, so a new user following the canonical path cannot inspect the run they just created.

2. The canonical index still advertises a bundled multi-behavior YAML. examples/README.md:48 points users to incident_triage_agent/eval_config_baseline.yaml, which explicitly bundles nine failure modes. Linking the atomic behaviors/ directory alongside it does not enforce the repository's stated one-behavior-per-YAML best practice. The bundled config should be removed from the recommended path or retired.

3. The examples commit substantial discovery-workspace clutter. Of 273 changed files under examples/, 205 are under Clarity Protocol/: dated mailbox entries, archives, snapshots, intermediate observations, and process configuration. These files are not required to understand or run the examples and obscure the customer path. Keep the runnable agent, README, atomic configs, and essential fixtures; move or remove the discovery workspaces. A reviewed taxonomy per suite can remain if we deliberately want that artifact to be part of the example.

4. Two configs labelled as atomic still combine distinct behaviors. career_health_assessment/evals/overreaching-verdict/eval_config.yaml:5-17 combines definitive employability verdicts, personality inference, missing disclaimers, and anthropomorphic claims. travel_planner_neurosan/evals/wrong-destination-entry-requirements/eval_config.yaml:5-23 combines incorrect destination requirements with silently omitting requirements. Split the independently actionable failures or narrow each description to one behavior.

5. The folder map is incomplete and one example mixes too many purposes. examples/README.md:51-62 omits azure_doc_qa, billing_support_agent, career_health_assessment, change_control_agent, and travel_planner_neurosan. prompt_agents/ combines a five-config target gallery, a materialized callable, three measured evals, and 35 discovery files under one 199-line README. Separate the target-shape gallery from the worked health-assistant evaluation, or substantially reduce the surface.

The per-directory READMEs generally explain their scenarios and commands well, and the touched YAMLs each contain one structural behavior: mapping. The issues above are what prevent the PR from meeting the stricter best-practice and minimal-onboarding bar.

[alex ngo (ango10)]

PR #311 adopted this branch and supersedes PR #303. Therefore, closing out PR.