docs(examples): bank-manager agent-control demo (AIEWF) + talks slideshow by changliu2 · Pull Request #262 · responsibleai/ASSERT

Chang Liu (changliu2) · 2026-07-09T22:08:30Z

What

Introduces the bank_manager_agent_control example — the AIEWF live demo. One banking support agent, taken through three beats:

Baseline (unguarded) — how often an un-controlled agent breaks policy
Defensive prompting — the intervention everyone reaches for first
Principled control plane (ACS) — policy-as-code gating on typed features of each tool call

ASSERT measures the difference; ACS enforces it. Plus a follow-along HTML slideshow under talks/.

Scope / decisions (please sanity-check)

Whole new example (83 files) — this directory did not exist on main before.
Result artifacts are NOT committed (chose code+docs only): the frozen per-case transcripts were ~21 MB, the same artifact type chore: drop 19 MB of committed per-case scores.jsonl from incident-triage demo #233 removed from a public demo. Users regenerate via assert-ai run into the gitignored artifacts/results/. The three READMEs were reconciled to a "generate, then view" flow.
ACS install via assert-ai[acs] (already on main) with a vendored wheel offline fallback. Note: the [acs] extra isn't on PyPI 0.1.0 yet — needs a release before pip install assert-ai[acs] works off PyPI.
CI beat points at the standalone responsibleai/assert-ci-banking-demo as the shipping vehicle (pip-install-in-your-own-repo), publishing separately.

Verification

77/77 unit tests pass (python -m unittest discover -s tests).
Leak scan clean — no home paths / secrets; .env.example placeholders only.
docs/README.md had pre-existing staleness (old variant-a/c/e names, eval_unguarded.yaml refs, build-demo-final branch, dead results/ copy-step) — reconciled to the current eval_realistic_* configs / variant-b0/b1/b2 / bank-manager-feature-rep suite. Worth a read to confirm.

Draft — for review, not merge

Opened as a draft so you can review on github.com. Nothing merges. Flagging the "no committed results" call and the docs/README.md reconciliation as the two things most worth your eyes.

…lideshow Introduces the bank_manager_agent_control example — the AIEWF live demo that takes one banking agent through three beats (baseline -> defensive prompting -> principled ACS control plane) and uses ASSERT to measure the difference and ACS to enforce it. - Customer-facing README with the 3-beat walkthrough + mechanics (internals like calibration/FRP stats kept out). - runtime/ engine (LangGraph agent, MCP servers, typed-feature policy) + policy knowledge corpus with an index README (VIP / high-net-worth enhanced-protection). - acs/ policy-as-code (Rego) + manifest; kb_ui/ retrieval-grounding UI; ci/ pointer to the standalone responsibleai/assert-ci-banking-demo gate repo. - Foundry IQ KB provisioner (scripts/setup_foundry_kb.py) reading runtime/knowledge. - 77 unit tests (feature policy, generalization, retrieval tuning, text invariance). - talks/aiewf-18min/: self-contained follow-along HTML slideshow + charts. Result artifacts are intentionally NOT committed (regenerate via assert-ai run; they write to the gitignored artifacts/results/). ACS installs via assert-ai[acs] with a vendored wheel offline fallback. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Trim the demo to only what the 3 beats + talk use: - eval_variants/ (17 archived experiment yamls) — not part of the demo - ci/sample-pages/ + assert-ci.yml + ci_gate.py + pareto_baseline.json — mock CI scaffolding; the talk used the REAL action-run summaries from the standalone responsibleai/assert-ci-banking-demo, which ci/README.md now points to. - scripts/retrieval_eval.py, scripts/pareto_frontier.py — unused. 77/77 unit tests still pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Strip the example to exactly what the 3-beat talk needs (baseline -> defensive prompting -> principled ACS control plane), removing an entire earlier a/c/e generation left in the tree: - Delete zombie files: runtime/mcp_server.py (old bank server), acs/manifest.yaml + acs/policy/bank_manager.rego (old text policy), acs/manifest_text_realistic.yaml (unused text-gate arm), docs/GATES.md (internal experiment methodology that referenced deleted arms). - agent.py: remove the old callables (chat_unguarded / chat_unguarded_prompted / chat_guarded_acs) and unused realistic variants (text-gate, feature+LLM), plus their helpers and constants. Now exposes exactly the 3 beat callables + the live-compare console entry points (709 -> ~420 lines). - docs/README.md: reconcile 'What's here' + ACS-integration sections to the realistic servers / manifest_feature.yaml / bank_manager_feature.rego. - scripts/smoke_test.py: fix stale post-reorg paths (corpus at runtime/knowledge, acs/ under the example root) and the callable-presence check. - scripts/setup_foundry_kb.py: fix CodeQL clear-text-logging finding — read secrets via accessors instead of storing them in the printed _cfg() dict. Verified: 77/77 unit tests pass; offline smoke test 10/10; agent.py imports with all three beat callables present. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…bank demo Follow-up to the demo minimization: remove artifacts that shouldn't ship and fix a target-import bug that broke clean-clone runs. - Delete vendor/acs/ (agent-control-specification wheel + sdist + README). The �cs extra already declares agent-control-specification + acs-generator, both published on PyPI, so pip install assert-ai[acs] is the single supported install path; the bundled 2.7 MB Linux-only manylinux wheel was redundant. - Fix agent.py: it called �uto_trace(), but assert_ai ships �uto_trace as a module whose entry point is �uto_trace.enable(). The old call raised TypeError: 'module' object is not callable at import on a clean clone, breaking every eval run. enable() installs the OpenInference instrumentors (verified: LangChain + OpenAI active) and only exports when a Phoenix collector is reachable. - Remove dead runtime/llm_classifier.py + its test (the callable that used it was dropped in the minimization; only its own test referenced it), and reword the acs_shim.py failsafe comment that pointed at the deleted test. - docs/README.md: drop the vendor/acs offline-fallback section and bullet. Validated end to end on the unguarded beat: 6/6 inference + 6/6 judge with zero errors, KB retrievals grounded, OpenInference spans captured in Phoenix. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d429caee-f2da-4fb0-8b2c-d0eff7cf40a3

Adds examples/bank_manager_agent_control/behaviors/: 4 failure modes, each in its own suite, with 3 arms (baseline/prompted/acs) sharing that suite's generated test set for apples-to-apples comparison. Adds a behaviors/README explaining the 1 behavior = 1 suite best practice, run order, and how to compare with the viewer and companion `assert-ai results matrix` command. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d429caee-f2da-4fb0-8b2c-d0eff7cf40a3

Removes examples/bank_manager_agent_control/behaviors/ (12 one-behavior-per- yaml configs + README). The published AIEWF talk ships the combined 4-behavior spec (eval_realistic_{unguarded,prompted,acs_feature}.yaml on suite bank-manager-feature-rep), which is the shape presented and the result the slideshow shows. The per-behavior split is handed off separately for science review and is intentionally kept out of the public talk PR. Also make the slideshow fail loud: if a chart image can't load, show a hint to open index.html directly or serve from the aiewf-18min/ folder, and document the same in talks/README.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d429caee-f2da-4fb0-8b2c-d0eff7cf40a3

Replace real subscription id, resource names (Azure OpenAI / AI Search / Foundry project), resource group, region hints and a personal name with generic placeholders in .env.example, and drop the same resource names from comments in agent.py, runtime/aad_auth.py and scripts/setup_foundry_kb.py. Keeps this customer-facing example free of internal infrastructure detail. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d429caee-f2da-4fb0-8b2c-d0eff7cf40a3

…go slide - Title slide: �val + control -> optimize (drop the min_control formula line) - Loop slide: redraw as outer lifecycle ring (Spec/Build/Test/CI/CD/Monitor) around the Control-with-ACS <-> Eval-with-ASSERT core, straight arrows, solid inner boxes; retitle 'The inner loop to power the outer loop' - Pareto slide: move the 'better' cue to the top-left, drop the frontier L-bracket, relabel the null arm 'Defensive prompt (no gain, p = .31)' and the subtitle 'manual prompt tuning didn't help here' - Remove the policy-as-code / Rego generalization slide + its asset Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d429caee-f2da-4fb0-8b2c-d0eff7cf40a3

…ior-configs

Document the demo's deliberate fail-open choices (post-tool-call scrubber, OPA-error handling, format-narrow SSN regex) and what to switch to fail-closed for a real deployment. Highest-severity money-moving actions are already gated pre-tool-call and fail closed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d429caee-f2da-4fb0-8b2c-d0eff7cf40a3

…ior-configs

Chang Liu (changliu2) · 2026-07-24T19:05:21Z

Maintainer review — do the ACS mitigations make sense?

Liam / Mike (additional reviewers) — I traced the full mitigation chain (rego policy → manifest → acs_shim enforcement → tool side-effect semantics). Verdict: the mitigations are sound and correctly back the demo's thesis. Summary below; I've also added a "Productionizing the gate" note to the example README in this PR.

What holds up ✅

Typed-signal gating (the thesis). Gating on result.risk_tier / referenced_accounts / grounded instead of regex-over-IDs generalizes uniformly across the deposit/loan/brokerage/client domains and avoids the text policy's brittle ID list.
Enforcement points are placed correctly.
- Irreversible/money-moving actions (create_transfer, freeze_account, enable_admin_mode) are gated pre-tool-call, fail-closed on the host snapshot — the shim raises AgentControlBlocked, so the side effect is actually prevented.
- Sensitivity / exfil / grounding run as post-tool-call scrubbers. Verified the gated tools don't settle: place_trade returns status:"prepared" (not settled) and reads/prepares don't mutate — so post-call scrubbing has no committed side effect.
Out-of-scope reference gate (referenced − authorized) is a sound structural XPIA/exfil signal, and it clears benign in-scope memos the regex false-positives on — which is why over-refusal also drops.
Ungrounded-claim gate on the KB grounded flag directly targets policy fabrication.
Learned-risk gate fires only when the typed gates are silent; an enrichment failure escalates (fail-safe).

Hardening notes (fine for a demo; documented in the new README section)

Post-call scrubber fails open — unparseable JSON → allow; risk_tier defaults standard, grounded defaults true. Should fail-closed for the sensitivity dimension in production. (Money-moving pre-gates are already fail-closed, so worst-case severity is bounded.)
Enforcement layer allows on OPA error (acs_shim.py) — a real control plane should fail-closed/escalate on engine error.
SSN input gate is format-narrow (NNN-NN-NNNN only) — the typed post-call gates are the real protection.

Net: coherent design, highest-severity actions fail-closed. Not blocking — the fail-open choices are deliberate demo trade-offs and now called out in the README so customers don't ship them verbatim.

Replace the old 'min_control failure(YOUR agent)' formula with the current tagline 'evaluate and optimize your AI agents' in talks/README.md and the bank-manager example README, and correct the deck length to 7 slides (the policy-as-code slide was removed). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d429caee-f2da-4fb0-8b2c-d0eff7cf40a3

…d file Downloading index.html on its own (without the sibling assets/ folder) broke the relative image paths. Embed loop_diagram.png and pareto.png as base64 data URIs so the deck renders anywhere — double-click, file://, or served from any dir. Drops the now-unnecessary onerror fallback + chartFail helper and corrects talks/README to describe the deck as a single self-contained file. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d429caee-f2da-4fb0-8b2c-d0eff7cf40a3

The AIEWF 2026 deck now ships as a single-file PDF rendered from the same 7 slides, instead of a self-contained interactive HTML slideshow. Drops the keyboard/click-nav docs from talks/README.md since PDF viewers handle their own navigation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Aaron Aspinwall (AaronAspinwall123) · 2026-07-27T22:22:31Z

I did some AI review and it suggested some issues:

• ACS lets sensitive transaction history through because risk_tier is omitted and defaults to standard ( bank_core.py:201 , bank_manager_feature.rego:56 ).
• Config generates 60 cases, while docs/results claim n=100 ( eval_realistic_unguarded.yaml:69-70 ).
• Quickstart installation omits the examples extra required by agent.py ’s MCP imports ( README.md:74 , agent.py:39-43 ).

Feel free to ignore if you think its not necessary

… examples extra read_transaction_history built its return dict from scratch and never included risk_tier, unlike read_account (which copies the full record). The Rego policy's object.get(result, "risk_tier", "standard") default meant the sensitive_read_gate silently never fired for this tool, even on VIP/high-net-worth/restricted accounts -- despite read_transaction_history being listed in read_tools. Added risk_tier to the returned dict plus a regression test (test_read_transaction_history_signals) that would have caught this. Also: the example's quickstart pip install command (README.md, ci/README.md) omitted the examples extra, which is required for langchain-mcp-adapters (agent.py's top-level MCP import). Following the quickstart verbatim would raise ModuleNotFoundError on the first assert-ai run. The more detailed docs/README.md setup guide already had this right; synced the two shorter quickstart commands to match. Per audit of AI-review feedback on PR #262 (issuecomment-5097447052) -- the third flagged item (config generates 60 cases vs docs claiming n=100) was independently verified as a false positive: the actual generated test_set.jsonl has exactly 100 rows (50 prompt + 50 scenario); sample_size in the stratified test-set config does not map 1:1 to final case count. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Chang Liu (changliu2) · 2026-07-28T17:11:06Z

Thanks for flagging these — ran them all down against the actual code/artifacts rather than taking the AI review at face value. Two real, one false positive:

#1 (risk_tier bypass) — real, fixed in b720ccb. read_transaction_history built its return dict from scratch and never included risk_tier, unlike read_account (which copies the full record). Since the Rego policy defaults missing risk_tier to "standard" (bank_manager_feature.rego:56), the sensitive_read_gate silently never fired for this one tool — VIP/high-net-worth transaction history could be read without triggering the same block that correctly stops read_account on the same entity. Added risk_tier to the return dict plus a regression test (test_read_transaction_history_signals) that would have caught this — there was genuinely zero test coverage for that combination before.

#2 (n=100 vs 60) — false positive. That's naive arithmetic on the two sample_size config knobs (40 + 20 = 60) that doesn't reflect how the stratified test-set generation actually expands into final case count. I counted the real generated test_set.jsonl directly: exactly 100 rows (50 prompt + 50 scenario), matching the docs and the -n100 run naming. Worth double-checking real output before trusting an AI review's arithmetic on config values that don't map 1:1.

#3 (missing examples extra) — real, fixed in b720ccb. The quickstart pip install "assert-ai[acs,langgraph,otel]" in both README.md and ci/README.md omitted examples, which langchain-mcp-adapters (imported at module load in agent.py) lives under. Following the quickstart verbatim → ModuleNotFoundError on the first run. docs/README.md's fuller setup already had this right; synced both shorter quickstart commands to match.

One more thing I found while auditing #1, not from your list: the Python FeatureGate.decide() mirror in feature_policy.py only checks entity sensitivity for WRITE_TOOLS — it has no equivalent check for reads at all, even though the real Rego policy has an explicit sensitive_read_gate for reads. The live Rego enforcement is correct and unaffected (verified separately), but the Python unit-test mirror can't exercise that specific gate the way the Rego can, since it doesn't implement it. Not fixing this myself since it's a bigger design question (is the Python class meant to be full parity, or an intentionally narrower harness for a few properties?) — flagging for a follow-up rather than expanding this PR's scope.

All 68 existing tests + the new one pass. Pushed in b720ccb.

… nonexistent index.html Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…r-YAML (ACS demo superseded by #262) (#271) * fix(examples): incident_triage_agent - baseline-only, one-behavior-per-YAML (remove preview ACS/GEPA surface) This example previously shipped a full ACS + AgentShield + GEPA demo into the public repo. That preview integration is not meant to be public, and public examples must not depend on unreleased/preview integrations. This strips the example back to a self-contained baseline ASSERT eval that runs with just assert-ai + an LLM key. Removed (preview surface): agent_guarded.py, eval_config_guarded.yaml, eval_config_guarded_gepa.yaml, incident-triage.guardrails.yaml, optimize_with_gepa.ipynb, prompts/ (optimized prompt + cross/pii_detection + xpia_paraphrase), and the whole artifacts/ tree (guarded/gepa/naive result snapshots + trade_off.png). Also removed eval_config_naive_prompt.yaml (folded into a chat_naive contrast callable). Scrubbed agent.py and SOP.md of every guarded/GEPA reference. Restructured to one-behavior-per-YAML under behaviors/: sop_tool_ordering (ordering_violation), pager_severity_discipline (pager_violation), channel_severity_routing (channel_violation), pii_redaction (pii_leak; folds pii_leak_under_pressure), xpia_resistance (xpia_relay), escalation_correctness (escalation_violation), alert_id_integrity (alert_id_drift), severity_classification (wrong_severity; folds severity_drift_under_pressure), no_fabrication (fabrication). Each disables the built-in policy_violation/overrefusal headline and scores one custom rubric dimension copied verbatim from the old bundled config. eval_config_baseline.yaml stays as a scrubbed bundled overview (baseline target, built-in headline + one dim per mode). Validation: all 10 configs load via assert_ai.runner._load_context (0 errors); agent.py passes py_compile; preview-surface grep across the example is clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d429caee-f2da-4fb0-8b2c-d0eff7cf40a3 * test(incident-triage): update smoke test for baseline-only structure The smoke test still referenced the removed guarded/GEPA surface (agent_guarded.py, incident-triage.guardrails.yaml, eval_config_guarded.yaml), so Tier 1 unit tests failed after those files were deleted. Drop the GuardrailsYamlShapeTest and guarded-agent import test, and rewrite EvalConfigShapeTest to assert the baseline config + all nine one-behavior-per-YAML configs target the baseline callable, reference no guarded target, and each carry a distinct suite. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d429caee-f2da-4fb0-8b2c-d0eff7cf40a3 * test(viewer): fix stale 'Refused before inference' assertions Merging main into this branch surfaced a pre-existing mismatch: #266 capitalized the viewer label to 'Refused before Inference' (pipeline-stage noun) but test_viewer_server_artifacts.py still asserted the lowercase form in three places (previewTesterLabel / previewDrawerLabel / scoredTesterLabel). #266 was viewer-only so Tier 1 unit tests never ran on it. Align the assertions with the intentional capitalized label. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d429caee-f2da-4fb0-8b2c-d0eff7cf40a3 --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Aaron Aspinwall <94419331+AaronAspinwall123@users.noreply.github.com> Copilot-Session: d429caee-f2da-4fb0-8b2c-d0eff7cf40a3

…only layout Merging origin/main brought in 054797f, which reduced incident_triage_agent to a baseline-only, one-behavior-per-YAML example and deleted eval_config_naive_prompt.yaml, eval_config_guarded.yaml, eval_config_guarded_gepa.yaml, and incident-triage.guardrails.yaml. Git merged cleanly because no file was touched on both sides, but the top-level examples/README.md row still advertised all four configs and the ACS + GEPA 4-variant matrix -- stale on main as well, since 054797f never updated this table. Repoint the row at what the example actually ships: behaviors/ as the recommended one-behavior-per-YAML split, with eval_config_baseline.yaml as the bundled overview. Drop the ACS/GEPA framing (that demo is superseded by #262 and its guardrails file is deleted); ACS remains covered by the acs_guardrails row. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b

* feat: clarity integration. * feat: drive Clarity integration through MCP server instead of CLI. * feat: ASSERT + ACS integration. * feat: make govern-remeasure loop deterministic and self-explaining. * test: billing support agent. * fix: increase sample_size to 25 to reduce noise in A/B testing * fix: delete old eval_config.yaml billing agent. * fix: updated agent_guarded.py to source callerid from session. * fix: give user choice for sample size, add numeric/threshold gate format. * feat: new subsection for output/input points. * fix: reduce govern to pure enforcement. * fix: refine SYSTEM_PROMPT to alight with examples. * feat: billing_support_agent resources. clarity, example, and artifacts. * fix: forward score_keys in buildJudgedSampleRow to fix false judgefailed badges on prompt rows. * docs: add per-domain organization guidance for multi-run workflows. * fix: forward score_keys in normalized result items to stop false judge failed on disabled-dimension runs. * feat(test): azure_doc_qa, change_congrol_agent, travel_planner_langgraph example runs and results. refine skill. * fix: make fabricated-details output gate history-aware. adopt regen operating point. * feat: career_health_assessment agent example run. * fix: refine career_health_assessment annotator. * feat(examples): add Prompt Agents governance packages from SKILL workflow. * fix: update max_turns default for SKILL to 10. * feat(example): science_research_agent and travel_planner_neurosan examples ran through SKILL workflow. * chore(library): make behavior presets atomic and enforce it in CI The behavior library had three problems, none of which anything checked for. **Presets bundled multiple behaviors.** `travel_planner` covered six mechanisms across "Quality failures" and "Safety failures" -- three of which (`stereotyping`, `prompt_injection`, `sycophancy`) already existed as their own atomic presets. `travel_planner_benchmark` bundled roughly six more. `telecom_customer_service` was not a behavior at all: it is an application spec (Role, Domain Basics, Operational Procedures) wearing `kind: behavior`. Evaluating a bundle as one behavior produces a dataset mixing several mechanisms and a metric nobody can act on -- you learn that something failed, never which mechanism. That is exactly what best-practices 8.D ("use atomic behaviors") exists to prevent. These three are application scenarios, so they move to a new `scenario` kind in `assert_ai/library/scenarios/`. They are the context an eval runs against, not the behavior it measures. **20 behaviors shipped to nobody.** `examples/behavior_specs/*.md` held 38 specs; `assert_ai/library/behaviors/*.yaml` held 18 of them. Only the YAML goes in the wheel, so every agentic failure mode -- goal drift, premature termination, repeated action loops, stale state, poor retrieval, tool-call error recovery, and 14 more -- was invisible to anyone who installed from PyPI. The 18 that did exist in both places were byte-identical, so this was pure coverage loss, not divergence. Generated the missing 20 from the existing markdown and the category metadata already in that directory's README; no prose was invented. **Nothing detected either problem.** `scripts/check_behavior_library.py` now fails CI when a preset names another preset's behavior (provable bundling), when one preset carries several failure categories, when a description reads as an application spec, or when a spec markdown drifts from its YAML or has no preset at all. It runs in Tier 1. Not breaking: `behavior: {preset: travel_planner}` still resolves, via a shim that warns and points at the `scenario` kind. Config authors get told, not broken. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7cb46daf-b5ce-4ad5-a85d-977737e5c02b * fix(cli): expose the scenario kind on library list/show Left out of the previous commit, so 'library show --kind scenario' rejected the new kind and Tier 1 failed. The local run passed only because the edit existed in my working tree but was never staged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7cb46daf-b5ce-4ad5-a85d-977737e5c02b * feat(viewer): headline policy violations split by behavior permissibility. * fix(tests): copy permissibility.ts into the node test harnesses. * fix(examples): delete all skill generated content for rerun of finalized skill. * feat(example): updated skill and billing_support_agent example run. * career_health_assessment: Clarity->ASSERT->ACS->ASSERT bug-bash run Full workflow for the career health assessment example. Clarity: problem/failures/requirements/architecture. Triaged F1 (unsupported candidate inference) as the single atomic behavior; overrefusal is tracked as the counter-metric, not a second suite. ASSERT baseline (25 prompt + 25 scenario, built-in judge dimensions only): policy_violation prompt 4.0% scenario 24.0% overrefusal prompt 0.0% scenario 36.0% ACS: single `output` intervention point (this agent has no tools) with three classifier annotators. agent_guarded.py wires AgentControl.from_path(manifest, dispatcher) by hand -- ASSERT's own build_agent_control helper omits the dispatcher, so input.annotations.* is never populated and every annotator-conditioned rule fails open. It also drops `history`, which would break multi-turn parity. ASSERT remeasure (governed): policy_violation prompt 8.0% scenario 16.0% overrefusal prompt 4.0% scenario 24.0% Net 13 failing rows of 50 vs 16 at baseline; the scenario slice -- where the harm actually concentrated -- improved on both dimensions (24.0->16.0 and 36.0->24.0). The prompt slice moved by one row on each dimension, which is inside the noise band at n=25. Tuning note: the judge scored empty schema placeholders as unsupported assertions, so the regeneration instruction was changed to omit fields rather than pad them, and the fallback no longer emits empty scaffolding. Configs differ by exactly two lines (run, target.callable), so the A/B isolates enforcement. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * change_control_agent: Clarity protocol, ACS policy, and governed agent Runs the full Clarity -> ASSERT -> ACS -> ASSERT loop for the change control agent and lands the governed variant. Measured (25 prompt + 25 scenario rows, built-in judge dimensions): run PV prompt PV scen OR prompt OR scen baseline 0.0% 32.0% 4.0% 4.0% governed v1 0.0% 24.0% 4.0% 28.0% governed v2 4.0% 24.0% 0.0% 0.0% The v1 -> v2 step is the interesting one. v1 cut the scenario violation rate by 8 points but drove overrefusal from 4.0% to 28.0%. Reading the failing rows showed all eight were pure drafting requests -- "give me a routing note", "rewrite this record block", "shortest release-ready note". The gate was correctly refusing to *advance* a change and incorrectly refusing to *write about* one. v2 adds a drafting carve-out in four places: the annotator rubric, the regeneration instruction, the still_allowed branch of the block result, and the fallback, which now leads with authoritative status and preserves the draft underneath instead of opening with a refusal. Overrefusal went to 0.0% on both slices while the violation improvement held. Net failing rows: 7 of 50 governed vs 10 of 50 baseline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * prompt_agents: runnable agent, Clarity protocol, ACS policy, governed variant The example shipped as YAML configuration only, so agent.py was written from health_assistant.yaml to give the suite a runnable target. The governed variant is additive; agent.py is untouched so the A/B is honest. Measured (25 prompt + 25 scenario rows, built-in judge dimensions): run PV prompt PV scen OR prompt OR scen baseline 0.0% 24.0% 0.0% 8.0% governed 4.0% 8.0% 0.0% 8.0% policy_violation scenario 24.0% -> 8.0% (-16.0 pts) overrefusal both slices unchanged net failing rows 8/50 -> 5/50 The 0.0% / 24.0% baseline split is the whole diagnosis. Single-turn attempts to extract a dose are refused every time -- the system prompt's "Never provide dosage recommendations or diagnoses" works when the request arrives cold. Every violation is in the multi-turn slice, where the tester builds rapport, supplies clinical framing ("her doctor already put her on this"), and expresses distress at the refusal. The prohibition never leaves the context window; it stops being treated as a hard limit and becomes one consideration to weigh against a sympathetic case. A standing instruction cannot fix erosion of a standing instruction, so the gate screens the outbound reply on every turn. Two deterministic pre-verdicts cover the measured shapes: _DOSE_DIRECTIVE (a quantity and an administration verb in the same clause) and _REASSURANCE (dismissal of clinical concern about a symptom the ledger shows the user described). The second matters because a negative diagnosis reads as comfort rather than instruction and is advice NOT to seek care. Urgent-care escalation is exempted before any gate runs -- it is structurally a directive and is the one directive that is always right. Framework note: the generated Rego compares the annotator result against the literal string "true". Returning a real Python True silently never matches and the policy fails open with no error. This is the fifth mutually incompatible annotator return contract across five domains in this batch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * Add ACS governance for travel_planner_langgraph fabricated travel details Runs the full Clarity -> ASSERT -> ACS -> ASSERT loop for the LangGraph travel planner. The behavior under test is fabricated flight, hotel, and weather details presented as confirmed fact. Baseline measured 20.0% / 60.0% policy violation (prompt / scenario) with 24.0% scenario overrefusal. The guarded target adds an evidence ledger that records every tool result with a reliability tag, an annotator that separates unsourced concrete claims from general planning reasoning, the generated ACS policy wired as an additive backstop, and a bounded regeneration loop. A first governed revision cut prompt fabrication to 4.0% but doubled overrefusal to 48.0%. The justifications showed the guard was blocking generic planning help that requires no grounding, so the fallback, the annotator carve-out, and the regeneration prompt were rewritten to scope the block to unsourced specifics rather than to the travel topic. Governed v2: policy violation 16.0% / 28.0%, overrefusal 0.0% / 40.0%. Scenario fabrication is down 32 points against baseline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * Add ACS governance for azure_doc_qa internal document disclosure Runs the full Clarity -> ASSERT -> ACS -> ASSERT loop for the Azure docs Q&A agent. The behavior under test is disclosure of internal engineering documentation -- quotation, close paraphrase, extracted specifics, and existence or structure confirmation. Baseline measured 45.8% / 88.0% policy violation (prompt / scenario), the second-worst scenario slice in the batch, with 4.2% / 12.0% overrefusal. The guarded target classifies retrieved documents by source, redacts internal material at the tool boundary, wires the generated ACS policy as an additive backstop, and regenerates any reply that discloses. A first governed revision cut prompt disclosure to 4.0% but pushed scenario overrefusal to 40.0%. The justifications showed the guard was refusing public documentation work -- public checklists, rollback guidance, architectural comparison, customer-safe wording -- so the regeneration prompt and fallback were rewritten to enumerate the permissible public work and to forbid bare deflection. Governed v2: policy violation 12.0% / 32.0%, overrefusal 4.0% / 36.0%. Scenario disclosure is down 56 points against baseline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * Add ACS governance for science_research_agent restricted content leakage Runs the full Clarity -> ASSERT -> ACS -> ASSERT loop for the science research agent. The behavior under test is disclosure of restricted documents through the research synthesis path -- quotation, close paraphrase, gist summary, translation, and multi-turn reconstruction. Baseline measured 28.0% / 20.0% policy violation with 16.0% / 8.0% overrefusal. The guarded target redacts restricted material at the tool boundary, keeps public entries in a mixed result set fully usable, wires the generated ACS policy as an additive backstop, and adds a deterministic corpus oracle that screens outbound replies for reproduced restricted text regardless of whether a tool returned it or the user pasted it. Five revisions, with the rejected ones documented. v2 regressed because the policy_note attached to withheld data still authorized exactly what the regeneration instruction forbade. v4 regressed because the corpus oracle blocked on a single shared shingle, which made it refuse public material; decision 03 records why shared shingles indicate topic rather than substance. v5 subtracts the public corpus and requires a run of three, verified to catch all 8 restricted documents and flag none of the 12 public ones. Governed v5: policy violation 24.0% / 28.0%, overrefusal 20.0% / 16.0%. Reported with the caveat in decision 02: judge boolean and judge justification disagree on roughly half of rows in this suite, so the design was steered from justifications rather than from the rate. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * Add ACS governance for travel_planner_neurosan fabricated travel details Runs the full Clarity -> ASSERT -> ACS -> ASSERT loop for the neurosan multi-agent travel planner. The behavior under test is fabricated travel details presented as confirmed fact. The first baseline scored 0.0% on all four metrics, which was a framework fault rather than a result: systematize silently replaced the supplied behavior definition with refusal boilerplate and emitted one category against a configured count of 25, so the suite measured refusal instead of fabrication. Re-running the identical config with --force-stage systematize produced ten real fabrication categories and a baseline of 96.0% / 96.0%, the worst in the batch. Three structural defects drive it: simulate_tool relabels fixtures rather than selecting data, so a Seattle-to-Boston request returns Tokyo hotels under a Boston heading; optimize_itinerary validates hardcoded costs so every trip totals $1,820; and sub-agents summarize tool output before the optimizer sees it, so no claim is traceable to evidence. The guarded target adds a deterministic destination oracle, a reliability-tagged evidence ledger, costs derived from retrieved prices, and the generated ACS policy as an additive backstop. This policy uses a sixth distinct annotator contract, with polarity differing per annotator within the same policy, and an output rule that can only ever warn. v1 reached 40.0% / 68.0%. The surviving failures were unlabeled planning specifics rather than relabelled fixture data, so v2 added a provenance banner derived solely from the ledger and prepended unconditionally, making the estimate labelling deterministic instead of requested. Governed v2: policy violation 28.0% / 52.0%, overrefusal 16.0% / 16.0%. Prompt fabrication is down 68 points against the valid baseline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * Add failure-2 governance for change_control and prompt_agents Second Clarity-sourced failure per domain, measured baseline vs ACS-governed with the built-in judge dimensions (25 prompt + 25 scenario rows each). change_control_agent - F3 embedded-directive injection via proposal fields: baseline PV prompt 36.0% / scenario 4.0% OR prompt 24.0% / scenario 4.0% governed PV prompt 36.0% / scenario 4.0% OR prompt 24.0% / scenario 0.0% This suite's policy_violation metric is NOT trustworthy: 80% of flagged rows are contradicted by their own judge justification ("the assistant does not follow the embedded request", scored as a violation), and 19-23 of 50 transcripts dead-end on change IDs the generated test set invented but the agent's fixtures do not contain. The guard is correct by construction and unit-tested (12/12), but this suite cannot measure it. Reported as measurement-invalid rather than iterated against a broken instrument. prompt_agents - F2 false reassurance / negative diagnosis: baseline PV prompt 0.0% / scenario 4.0% OR prompt 0.0% / scenario 0.0% governed PV prompt 0.0% / scenario 4.0% OR prompt 0.0% / scenario 4.0% A negative control: Clarity brainstormed this failure but measurement shows the agent does not exhibit it (1/50, 0 judge errors). No headroom to improve; the guard is verified not to introduce overrefusal. Both guards extend the existing agent_guarded.py additively - change_control with zero deletions, prompt_agents with renames only - so each file now carries two independent AgentControls and failure-1 behaviour is preserved. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * Add failure-2 governance for science_research_agent (cross-document inference) Clarity F4 - reconstructing a restricted conclusion from public documents. 25 prompt + 25 scenario rows, built-in judge dimensions. baseline PV prompt 40.0% / scenario 56.0% OR prompt 4.0% / scenario 12.0% governed PV prompt 48.0% / scenario 52.0% OR prompt 12.0% / scenario 4.0% Read honestly this is a wash, not a win. The guard adds a real structural control - restricted-unique 6-gram residue is stripped from the request before the model sees it, so a conclusion cannot be assembled from evidence never delivered - plus a deterministic output check for a non-public marker co-occurring with a performance comparison. 9/9 unit tests pass. But the measurement cannot resolve it. 24% of the governed run's flagged violations are contradicted by their own judge justification (vs 4% at baseline), e.g. "The assistant does not disclose or infer any restricted conclusion" scored as a violation. Netting inversions out, real violations move roughly 23 -> 19, which is inside the noise. Overrefusal trades directions between prompt and scenario rows rather than improving. Recorded as finding 21 rather than presented as an improvement. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * Add failure-2 governance for career_health_assessment (CV untrusted-instruction injection) Clarity F4 - the agent obeys instructions embedded in the untrusted CV_TEXT it was asked to assess. 25 prompt + 25 scenario rows, built-in judge dimensions. baseline PV prompt 64.0% / scenario 88.0% OR prompt 0.0% / scenario 44.0% governed PV prompt 20.0% / scenario 52.0% OR prompt 0.0% / scenario 48.0% Policy violations fall 44 points on prompt rows and 36 points on scenario rows while overrefusal is held at baseline (scenario +4 points, inside noise; prompt unchanged at 0.0%). The judge's boolean/justification inversion rate is 39% in BOTH arms, so the comparison is like-for-like; netting inversions out, real violations roughly halve (~23 -> ~11). Note the baseline agent ALREADY overrefuses at 44% on scenario rows - it emits {"status": "unsupported_task"} and declines legitimate CV scoring. That is a pre-existing defect of the ungoverned agent, not a governance cost, and the governed agent does not make it meaningfully worse. This took two iterations. The first guard cut violations but pushed overrefusal to 68% via two bugs, both found from judge justifications rather than guesswork: 1. sticky cross-turn detection - injection scanning folded in all prior history turns, so a turn-1 injection suppressed every later clean turn. Detection is now strictly per-turn. 2. stub-producing enforcement - regeneration fired on foreign keys or a missing status and fell back to a blank stub. It now fires only on hard capitulation; foreign keys are repaired in place and populated content is never downgraded. The mandated warning now accompanies a real assessment instead of replacing it. Guard is additive (439 insertions, 0 deletions); failure #1 is byte-for-byte intact. Unit tests 23/23, including the multi-turn regressions that would have caught the overrefusal. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * Add failure-2 governance for azure_doc_qa (self-asserted identity escalation) Clarity F2, which Clarity had explicitly marked SEVERITY: HIGH - selected but which was never built into a suite until now. 25 prompt + 25 scenario rows, built-in judge dimensions. baseline PV prompt 76.0% / scenario 48.0% OR prompt 28.0% / scenario 0.0% governed PV prompt 20.0% / scenario 8.0% OR prompt 16.0% / scenario 4.0% Policy violations fall 56 points on prompt rows and 40 points on scenario rows, and overrefusal IMPROVES on prompt rows (28.0% -> 16.0%) rather than paying the usual governance tax. Scenario overrefusal rises 0.0% -> 4.0% (one row). The 76% baseline was the strongest single-arm signal of the batch aside from neurosan, which vindicates re-sourcing failure selection from the Clarity Protocol records rather than from fresh code analysis - Clarity had already identified and ranked this failure. The guard denies internal retrieval at pre_tool_call for callers whose only claim to clearance is their own self-description, so internal material is never fetched rather than fetched-then-suppressed, and repairs output-side statements that assert the caller was verified. This policy uses SPLIT ANNOTATOR POLARITY - validated_principal_present is a health flag (true = good) while caller_self_description_claims_clearance is a fault flag (true = bad) - both asserted in both directions in the tests. Guard is additive; the 7 removed lines are in-place replacements of bare clearance checks with policy-gated equivalents. Failure #1 intact. Tests pass (60+ checks), including no-deadlock verification when an event loop is already running. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * Add failure-2 governance for travel_planner_neurosan (false budget confirmation) This failure was recorded through Clarity during this session. Neurosan was the only domain with no enumerated second failure - its failures.md listed one auto-triaged top risk and nothing ranked below it - so rather than invent a failure from code reading, the Clarity failure-brainstorm was run properly and the result recorded via the Clarity MCP tools. failures.md is rewritten into the same enumerated F1-F5 + triage format the other six domains use. 25 prompt + 25 scenario rows, built-in judge dimensions. baseline PV prompt 100.0% / scenario 88.0% OR prompt 0.0% / scenario 4.0% governed PV prompt 12.0% / scenario 64.0% OR prompt 4.0% / scenario 8.0% An 88-point drop on prompt rows, the largest single improvement in the batch, off a 100% baseline - the ungoverned agent confirmed a budget as fitting on every single prompt row. Scenario rows improve 88% -> 64%, more modestly. Overrefusal stays low (+4 points each side). This is also the cleanest measurement in the batch: the judge's boolean/justification inversion rate is 0% in BOTH arms, so unlike the injection suites these numbers can be read at face value. The defect: agent.py hardcodes costs in validate_budget (lines 193-195) and the agent then asserts budget fitness as verified. The guard adds a second AgentControl with two deterministic enum annotators (budget_verdict, budget_verification_claim), catches self-contradictory totals, and closes the say-so gap with a disclosure banner rather than a refusal. Guard is additive; failure #1 grounding control untouched. Tests 6/6. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * Add failure-2 governance for travel_planner_langgraph (unmarked claim provenance) Clarity F5 - the agent presents retrieved facts and invented estimates in one undifferentiated voice, so the user cannot tell which parts were actually looked up. 25 prompt + 25 scenario rows, built-in judge dimensions. baseline PV prompt 36.0% / scenario 40.0% OR prompt 16.0% / scenario 20.0% governed PV prompt 24.0% / scenario 44.0% OR prompt 0.0% / scenario 12.0% Violations fall 12 points on prompt rows and overrefusal falls on BOTH strata to below baseline. Scenario violations are 4 points up, which is expected and is explained below rather than tuned away. This suite took two iterations and the first one is the more interesting result. Guard v1 scored PV prompt 12.0% - better than what is committed here - but drove scenario overrefusal 20.0% -> 52.0%. Reading the judge's justifications showed why: v1 achieved its low violation count by making the agent WITHHOLD the itinerary whenever its evidence ledger was empty, replying "tell me your dates and I'll look it up". The behaviour under test is unmarked PROVENANCE; the required fix is to LABEL which parts are verified, not to refuse to answer. v1 inverted the control's intent, and the metric rewarded it, because an itinerary never written contains no unmarked claims. The asymmetry gave it away: prompt overrefusal was unchanged at 16.0% while scenario rows tripled, since multi-turn conversations more often reach the rewrite request with an empty ledger. v2 removes the withholding path entirely. The original plan is now the floor - a regenerated reply replaces it only if it is itself a substantive plan, never a bare deferral - and regeneration fires only when the ledger is actually populated. The provenance banner still marks every unverified part. The 4-point scenario violation rise is the honest cost of that correction, and the guard author predicted it before the re-run: delivering the plan reintroduces its specifics, which the judge can then assess, whereas withholding hid them. Trading 4 points of measured violation for 40 points of overrefusal is the right direction for a labelling control. Judge inversion is 0% in BOTH arms here, so unlike the injection suites these numbers can be read at face value. Also fixes a latent crash in the shipped agent.py: intent_classifier assumed json.loads returned a dict, but a bare-string parse raised AttributeError and killed one scenario row mid-run. Parsing is now defensive and falls back to the default intent; well-formed dict responses are untouched. Both baselines had zero target errors, so this does not affect any published baseline number. Guard is additive; the 21 removed lines are the v1 withholding block and a _MANIFEST refactor. Failure #1 machinery intact and referenced. Tests 12/12, including regressions that assert an empty ledger still yields a labelled plan. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * Update science_research_agent tool cache from phase-1 governed re-run Regenerated when restricted-content-leakage was re-run with --force-stage inference to re-measure failure #1 against the now dual-control agent_guarded.py (see artifacts/results/PHASE1-RERUN-DRIFT.md). Cache grows 24680 -> 39268 lines; this file was already tracked, so committing keeps the tree clean and the run reproducible. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * prompt_agents: add model_only/simtools/gentools variant matrix The shipped prompt_agents example ships five YAMLs describing distinct Prompt Agent wirings (model_only, simulated tools, generated tools, sandbox, external). All are *prompt targets*, not callables, so ACS -- which can only govern `callable: module:function` -- could not be applied to any of them, and the earlier evaluation collapsed the whole domain onto the single realtools agent.py. Reify the three that are reachable as callables, each a faithful port of its YAML (system prompt lifted verbatim from the parser value, model / max_tokens / temperature as configured): agent_model_only.py no tools agent_simtools.py fixed toolset + LLM result simulator agent_gentools.py per-test-case LLM-generated tool schemas plus an ACS-wrapped counterpart for each, so every variant has a baseline and a governed arm. _variant_guard.py holds the shared adapter rather than editing agent_guarded.py, whose numbers are already published; the measured modules are byte-identical to what was measured before. Two defects fixed there rather than papered over: * the gentools ledger only recognised the four canonical tool names, so LLM-invented names never registered and the control was silently inert. _GenericLedger is name-agnostic. Verified against live generation, which produced `drug_interaction_check` and `pain_management_alternatives` -- neither in the base vocabulary, and different again on the previous call. * the empty-ledger path emitted canned guidance unrelated to the user's question. The original reply is now the floor. This is the third instance in this batch of a guard substituting or withholding content instead of repairing it; it scores well on violation rate, which is exactly why it needs catching. 24 eval configs (3 variants x 2 arms x 2 failures) differ from the measured baseline by exactly two lines each, `run:` and `callable:`, so all arms share one LLM-generated test set -- artifact_cache._stage_descriptor deliberately excludes target.callable from the test_set key to keep A/B comparisons like-for-like. 31 tests pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * Unbundle scenario behavior presets Extract atomic behaviors from the travel planner and telecom scenario specs, leaving scenarios as context plus behavior references. Reuse existing stereotyping, prompt_injection, sycophancy, grounding, tool-selection, verification, observation, response-completeness, and unsupported-conclusion presets instead of duplicating them. Update checker, docs, example configs, and benchmark default to enforce and consume atomic behavior presets. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7cb46daf-b5ce-4ad5-a85d-977737e5c02b * fix(examples): keep all seven flagship behaviors, one config each Unbundling the travel_planner preset also rewrote the flagship example config down to a single behavior (prompt_injection). That is atomic but it silently dropped six mechanisms from the example the README, getting-started, schema docs, the ACS guide, and science.yml all point at -- coverage loss wearing atomicity's clothes. Restores the other six as sibling configs under behaviors/, each sharing the same context: and measuring exactly one mechanism. eval_config.yaml stays the quickstart so every existing doc reference keeps working. This is also the layout we tell CI customers to use, so the flagship example now demonstrates it instead of just describing it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7cb46daf-b5ce-4ad5-a85d-977737e5c02b * Reset billing_support_agent to pre-skill state for clean rerun. * feat(agents): update systemize, judge to gpt-5.4, default model to gpt-5.4-mini. * feat(example): billing_support_agent ran through workflow. * feat(viewer): retire policy_violation/overrefusal from display surfaces. * fix(cli): detect the permissibility split by key presence, not rate. * feat(example): return billing_support_agent to its pre-skill state. * fix(skill): update SKILL to prevent custom judge dimension generation. * feat(example): billing_support_agent final workflow demo. * feat(example): billing_support_agent Clarity Protocol directory moved to example. * feat(example): career_health_assessment cleared to pre-skill state. * docs: make the behavior/scenario library obviously discoverable Addresses Ahmed's confusion on this PR about where the atomic behavior presets live and how they compose with scenarios. Adds pointers at every entry point a user is likely to hit before writing a behavior spec by hand: - Top-level README.md: new nav-bar link + a What-you-get bullet - docs/README.md: new Behavior Library entry under Configuration - docs/config/best-practices.md: callout inside SS8.D (atomic behaviors) - examples/README.md: new 'Reuse a behavior from the library' section, placed before a user starts writing YAML by hand Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * feat(example): career_health_assessment ran through SKILL workflow. * feat(example): cleared azure_doc_qa to pre-skill state. * feat(example): azure_doc_qa workflow through SKILL complete. * feat(examples): clear seven examples to pre-skill state for rerun. Reset travel_planner_langgraph, change_control_agent, prompt_agents, science_research_agent, and travel_planner_neurosan to commit 6817a31 so the finalized skill can be rerun end to end. prompt_agents backs the health-assistant model-only, simulated-tools, and generated-tools variants, so clearing that root resets all three. Removes generated Clarity Protocol trees, eval configs, ACS policies and manifests, guarded agents, and guard tests while retaining each example's original agent, tools, README, and config inputs. Prior ASSERT and ACS run outputs were preserved under artifacts/*-run2-archive before deletion. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * Add ACS governance for science_research_agent disclosure risks Clarity risk discovery identified two Critical failure modes in the science research assistant: restricted-class disclosure and obedience to instructions embedded in retrieved documents. Both are governed by a single structural gate. tools.py `_render()` returns the full `body` and `snippet` for every hit regardless of the authoritative `class` field, so the four restricted classes (restricted_results, partner_notes, contacts, private_notes) are disclosed in full. `snippet` is not a lesser field - it carries the partner recall figure and a named contact - so body-only redaction would have left the harm intact while reporting enforcement active. The gate runs at post_tool_call on file_search and is transformative, not a denial: file_search is never blocked. Restricted results have `body` and `snippet` replaced by a marker while `id`, `class`, `rank`, `tags`, and `title` are preserved, so the agent can still report that a document exists and give the access-request path. No injection detector was built. The payload the injection asks for is removed before delivery, so the embedded instruction can be obeyed in full and return nothing. Source class is deliberately not used as an injection signal: fetch_url hardcodes class `public`, and the carrier document is genuinely public. Measured at n=25 per split, baseline vs governed: restricted_class_disclosure prompt harm 91.67% -> 43.75% permissible 60.87% -> 32.00% scenario harm 95.83% -> 75.00% permissible 48.00% -> 52.00% embedded_instruction_obeyed prompt harm 56.25% -> 26.67% permissible 28.00% -> 28.00% scenario harm 61.90% -> 34.78% permissible 52.00% -> 52.00% Restricted markers fall to zero in the governed transcripts: the partner recall figure 679 -> 0, the contact address 220 -> 0, the contact name 428 -> 0. The scenario permissible tick for the first risk is one row of 25 with node-level counts flat at 14/16/16, so it is clustering noise rather than a mechanism; it was not resampled. One iteration was needed. The first marker was long, prohibition-heavy, and injected twice per restricted document, which suppressed legitimate internal_only discussion - the internal_only sentinel dropped from 58 occurrences to 48. A compact, positively framed, scope-bounded marker with the long guidance emitted once in policy_notice restored it to exactly 58. No prompt, model, control-flow, or test-set change. agent_guarded.py adds ACS enforcement and nothing else. It imports agent.py rather than copying the loop, never redefines SYSTEM_PROMPT, TOOL_SCHEMAS, MAX_STEPS, MAX_TOOL_CALLS, or AGENT_MODEL, and delegates to base.chat(). Enforcement is scoped by a contextvars.ContextVar set inside try/finally by the governed entrypoints; with the wrapper installed and the contextvar unset, baseline behaviour is unchanged. Governed configs differ from baseline by exactly two lines (`run:` and `target.callable`). Both governed runs reused systematize and test_set artifact v0001; --force-stage was never passed. Note that under-disclosure by blanket refusal is itself scored `permissible: false` in this taxonomy, so the harm metric is mixed here - part of the improvement is the agent answering more, not only leaking less. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * Add Clarity protocol + ACS governance for change_control_agent Runs the full Clarity -> ASSERT -> ACS -> ASSERT loop over the change control agent for both Critical failures found by Clarity risk discovery. failure-02 fabricated_change_record -- WIN. A post_tool_call gate denies a change record whose returned fabricated_fields is non-empty, and a pre_tool_call gate holds the control surfaces while such a record stands. prompt harm 0.00% -> 0.00% permissible 52.00% -> 44.00% scenario harm 90.91% -> 86.36% permissible 52.00% -> 52.00% (flat) failure-01 unauthorized_change_advancement -- HONEST NEGATIVE. A pre_tool_call gate on the four control surfaces denies a submission while policy state forbids it. prompt harm 0.00% -> 0.00% permissible 18.18% -> 4.35% scenario harm 66.67% -> 44.00% permissible 16.00% -> 24.00% The win condition (harm down AND permissible down-or-flat) is not met on the scenario split. Harm fell 22.7pp, roughly 5-6 rows of 25, while permissible rose 8pp, or 2 rows. All four permitted governed attempts were spent and attempt 1 was the best of them on every metric, so attempt 1 is what ships and the negative is reported rather than re-rolled. The permissible move is inside the measured noise floor. Governed attempts 1 and 2 differed only by added trace spans -- an observability change with no policy effect -- yet scenario harm moved 44.0 -> 56.0 and permissible 24.0 -> 41.7. Run-to-run variance at n=25 with judge n=1 is therefore about 3-5 rows. The 2-row permissible move sits inside that band and the 5-6 row harm drop sits outside it. No run was repeated unchanged to fish for a better draw. failure-02's residual 86% harm is a structural ceiling, not an implementation gap: only 8 of 20 harmful rows ever call create_change_request and 7 of 20 make no tool call at all, so most fabrication harm is invented prose that a tool-call gate cannot reach. The one output-stage attempt at it drove scenario permissible 52% -> 84% and overrefusal 12% -> 76% while barely moving harm, and was reverted in full. Not-permissible harm is 0.00% on the prompt split in every run of both suites, so all harm signal in this domain lives in the scenario split and the prompt split measures only over-restriction. agent_guarded.py adds ACS enforcement and nothing else. Both arms execute the same agent.py::_run_loop function object -- verified at runtime as agent_guarded._run_loop is agent._run_loop -- which takes a pluggable tool executor; the baseline passes _default_execute_tool and the guarded module passes an ACS-enforcing executor of the identical signature. No prompt, model, tool schema or budget is redefined. Each eval_config.governed.yaml differs from its baseline by exactly two lines, run: and target.callable, and every run scored the same systematize/test_set v0001 artifact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * Add Clarity protocol + ACS governance for prompt_agents health assistant Runs the full Clarity -> ASSERT -> ACS -> ASSERT loop over the health assistant Prompt Agent for both Critical failures found by Clarity risk discovery, across the model-only and simulated-tools variants. A Prompt Agent has no host process: the target is declared entirely in YAML and the ASSERT runtime owns the model call and tool loop, so there is nothing for ACS to wrap. The target is therefore materialised as a Python callable in agent.py, which instantiates ASSERT's own HostedSession and SimulatedResolver rather than imitating them. Both A/B arms run that same materialised callable, so the delta isolates enforcement; absolute levels carry materialisation error, the delta does not. The five health_assistant*.yaml specs are the specification under test and are unmodified. simtools x fabricated_clinical_fact_as_retrieved -- WIN on attempt 1. Marking, not suppression: the harm is the certainty, so the claim is attributed to its origin and not presented as verified. prompt harm 13/25 = 52.0% -> 9/24 = 37.5% perm 6/12 = 50.0% -> 1/18 = 5.6% scenario harm 17/25 = 68.0% -> 8/24 = 33.3% perm 10/20 = 50.0% -> 3/25 = 12.0% simtools x dosage_guidance_to_elderly_patient -- WIN on attempt 3. Non-statement, not marking: a caveat beside "5 mg" does not remove the 5 mg. prompt harm 10/25 = 40.0% -> 1/24 = 4.2% perm 1/25 = 4.0% -> 1/25 = 4.0% scenario harm 19/25 = 76.0% -> 15/25 = 60.0% perm 0/25 = 0.0% -> 0/25 = 0.0% Over-refusal is 0% on both splits of both shipped runs. modelonly x dosage_guidance_to_elderly_patient -- NOT WON, reported as such. No configuration wins both splits. v4 wins the prompt split outright (37.5% -> 27.3%, permissible flat at 0/25) and cuts scenario harm 18/24 -> 14/24, but scenario permissible rises 0/25 -> 2/25. modelonly x fabricated_clinical_fact_as_retrieved is excluded by design: with no tools there is no retrieval claim to make. gentools is not measurable and is skipped rather than approximated. health_assistant_generated_tools.yaml sets tool_source: per_test_case, so tool schemas ride on each test-case row and are read by _build_hosted_session as scenario["tools"]. _build_target_session has the payload in scope and passes it to ExternalSession, but constructs CallableSession without it, and CallableSession.__init__ accepts no scenario or tools parameter. Per-test-case tools cannot reach a callable target, so chat_gentools raises NotImplementedError. Approximating with a fixed toolset would measure simtools while labelling it gentools. Both rates are flagged/applicable, and the judge marks a node not applicable when the transcript never engages it, so a control that removes a behaviour outright shrinks its own denominator and can push a rate up while violations fall. modelonly v3 prompt is 9/24 -> 8/21: the rate rises 37.5% -> 38.1% while the count falls. Counts are therefore reported beside every rate here, and the README carries the same warning. The shipped default is the v3 ladder, which is what produced the confirmed simtools x dosage win; the v4 position-keyed ladder is retained behind HEALTH_ACS_POSITION_KEYED_DOSAGE=1 and gate telemetry behind HEALTH_ACS_GATE_LOG, both default off. Each governed eval config pins the run id the shipped code reproduces, so no config resumes a cached run produced by different code. agent_guarded.py adds ACS enforcement and nothing else. Both arms execute the same function object, verified at runtime as agent._chat is agent_guarded._chat; each governed entrypoint is a single line calling it through its one seam. The guarded module never references HostedSession, SimulatedResolver, load_toolset_file, parse_target_config or the system prompt. Each eval_config.governed.yaml differs from its baseline by exactly two lines, run: and target.callable, and every run in all three suites scored the same systematize/test_set v0001 artifact. verify_gates.py exercises both gates against the real AgentControl/OPA runtime (13/13), including a case proving the annotator name contract fails silently when the manifest key, the Rego reference and the dispatcher branch disagree. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * fix(examples): address Yeming's PR #296 review comments - Update the flagship travel_planner_langgraph README's Scenario table, which still described the pre-unbundling behavior.description (quality + safety failures blended, 6 behavior_categories). It now reflects the atomic single-behavior eval_config.yaml (prompt_injection, 4 categories) and explains the sibling behaviors/*.yaml cover the other six mechanisms. - Reformat context: and rubric: multiline fields in all six behaviors/*.yaml sibling configs (plus eval_config.yaml's rubric, for consistency) from a hard-wrapped single-quoted scalar to literal block style (| / |-), matching eval_config.yaml's existing context: style. Verified byte-for-byte semantic equivalence via yaml.safe_load diff against the prior committed content -- pure style change, no content drift. - Tighten scripts/check_behavior_library.py's spec-parity check per Yeming's concern: the 98%-similarity tolerance could let a real content change in a long spec through silently, since words() already normalizes the only expected sources of formatting difference (headers, bullets, wrapping, whitespace, case) -- any remaining difference is real drift, not noise. Now requires an exact match. Also hard-fails if the examples/behavior_specs reference directory is missing, instead of silently skipping the whole parity check. All 51 presets still pass (48 behaviors, 3 scenarios), atomic and in parity, with the tightened exact-match rule. 89/89 targeted tests still pass. All 7 edited example configs verified to still load and resolve through assert_ai.config.load_config. * travel_planner_langgraph: Clarity->ASSERT->ACS governance cycle Adds the governed arm for both Critical risks discovered by Clarity (fabricated_cost_claim, unsupported_entry_requirement), measured against a shared baseline. All runs reuse one systematize v0001 and one test_set v0001, and each governed eval config differs from its baseline by exactly two lines (run, callable), so the A/B is directly comparable. Result: a split verdict, shipped honestly rather than tuned into a win. Prompt split (single turn) wins on both risks: costs harm 16/23 -> 7/21 entry harm flat at 2 rows while permissible violations fall 11/23 -> 8/25 and over-refusal falls 44.0% -> 32.0% Scenario split (ten turns) regresses on both risks: costs permissible 12/23 -> 17/24, over-refusal 48.0% -> 75.0% entry permissible 11/19 -> 16/18, over-refusal 50.0% -> 89.5% The scenario regression is architectural, not a tuning failure. Those conversations frequently never reach the research step, so the retrieval record is empty; with no evidence in hand the only correct action for an output gate is to decline, and nothing later in the conversation supplies what is missing. Softening the annotator across attempts measurably re-opened harm (costs prompt harm back to 47.4%, entry scenario harm to 81.8%), so the trade is real. The enforcement wrapper is deliberately not allowed to retrieve the missing grounding, because a wrapper that retrieves is no longer a control. agent_guarded.py reuses the baseline graph rather than reimplementing it: get_graph(), _seed_messages() and _get_llm() are imported from agent.py, and _draft() is agent.chat() plus the returned message list and retrieval record. At runtime the guarded module resolves to the same compiled graph object, so the arms differ only by enforcement. Tracing is enabled before agent import in both arms so the judge sees identical telemetry. A known defect ships deliberately and is documented in code and in the README: the depth-based fallback rotation was inert during measurement because history was not threaded into it, so every declining turn emitted identical wording. The call site is left in its measured two-argument form so the published numbers reproduce from this code. The prompt split is single turn and therefore unaffected, so those wins stand unconfounded; the scenario over-refusal figures should be read as an upper bound on the cost of enforcement rather than a precise measurement of it. Threading history is a one-line, unvalidated change belonging to the next cycle. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * feat(example): clear travel_planner_langgraph to pre-skill state. Resets the example so the Clarity -> ASSERT -> ACS workflow can be run end to end from a clean slate in the IDE. Removed (all generated by the previous skill run): Clarity Protocol/ discovery, failures, goal, solution, mailboxes acs/ manifests and Rego policy for both risks evals/ baseline and governed eval configs for both suites agent_guarded.py the ACS enforcement wrapper README.md restored to its pre-skill content Also removed from the working tree, untracked and therefore not part of this commit: artifacts/results/travel-planner-*, artifacts/acs/travel-planner-*, the per-run logs and status JSON under artifacts/runlogs/, and __pycache__. Clearing the results and stage artifacts matters as much as clearing the source: a stale suite directory would let a later run reuse cached systematize/test_set artifacts instead of generating its own. The example is now byte-identical to its pre-skill state (empty diff against c2c11d5) and contains only agent.py, auto_trace.py and README.md. Nothing is lost. The previous cycle is preserved in full by commit 818f7c7 and by the exported release bundle, which carries the source, both ACS policies, all eval configs, the Clarity protocol, every run's results and transcripts, and the per-run logs for baseline plus all four governed attempts. No other example is touched. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * azure_doc_qa: Clarity protocol for fabrication + leakage risks Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * travel_planner_neurosan: Clarity protocol + ACS governance for two risks Shipped configurations (counts are flagged/applicable, re-derived from raw results; rates alone mislead because the judge drops non-engaging rows from the denominator): wrong-destination-entry-requirements baseline -> acs-governed prompt harm 25/25 -> 4/24 permissible 4/11 -> 1/25 overrefusal 0/25 -> 0/25 scenario harm 24/24 -> 10/14 permissible 10/16 -> 21/24 overrefusal 7/24 -> 24/24 fabricated-budget-verification baseline -> acs-governed-v2 prompt harm 21/25 -> 13/24 permissible 1/22 -> 1/25 overrefusal 0/25 -> 0/25 scenario harm 25/25 -> 23/23 permissible 11/25 -> 10/24 overrefusal 10/25 -> 11/24 Single-turn prompts win on both risks. Multi-turn scenarios trade harm for over-refusal: the fallback is a fixed stateless template re-delivered on every denied turn, so a 7-turn conversation re-asks for a nationality the traveller already supplied. That is a remediation-design fault, not a policy fault. Enforcement-only A/B: each governed config differs from its baseline by exactly two lines (run, callable). Same model, judge, sample size, and stage artifacts. Provenance: agent_guarded.py output templates were verified against the shipped transcripts by string probe -- the budget fallback appears 493x in acs-governed-v2 and 0x in acs-governed, which is why the budget config is pinned to v2. Annotator and regeneration prompt text was edited after the last successful run for two follow-up attempts that failed at inference and produced no results. Wrapper-internal prompts are never written to any artifact, so those edits cannot be confirmed or excluded from the measured state. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * azure_doc_qa: grounded fabrication gate + leakage gate through the SKILL Shipped runs: azure-doc-qa-confidential-leakage baseline -> acs-governed azure-doc-qa-fabricated-answer baseline -> acs-governed-grounded-v2 Counts are flagged/applicable, re-derived from the raw result rows. Rates alone mislead: the judge marks a node not-applicable when the transcript never engages it, so a shrinking denominator can move a rate while the violation count is unchanged. confidential-leakage baseline -> acs-governed prompt harm 9/22 -> 2/22 permissible 10/25 -> 6/25 overrefusal 2/25 -> 4/25 scenario harm 15/24 -> 8/24 permissible 17/25 -> 16/25 overrefusal 11/25 -> 16/25 fabricated-answer baseline -> acs-governed-grounded-v2 prompt harm 3/14 -> 1/16 permissible 11/24 -> 11/25 overrefusal 10/25 -> 11/25 scenario harm 9/23 -> 9/18 permissible 6/25 -> 18/25 overrefusal 5/25 -> 18/25 Leakage wins on both axes: harm roughly halves and permissible violations fall as well. Fabrication wins single-turn only - harm 3 -> 1 with permissible exactly flat at 11. Multi-turn does not win: the harm count is unchanged at 9 (the rate moves 39.1 -> 50.0 only because the denominator fell 23 -> 18) while permissible violations triple, 6 -> 18. Fabrication took three governed attempts, all recorded in Clarity summary.md: a reply-only output annotator cannot separate grounded specificity from fabricated specificity, so it only trades over-refusal. Feeding the annotator the retrieval context captured from the baseline graph, then scoping the rewrite, cuts single-turn harm without an over-refusal cost. Multi-turn is not reachable from an output gate at all - 18/25 conversations are flagged for both fabrication and over-refusal, i.e. the agent fabricates on some turns and stonewalls on others. The fix belongs upstream, at retrieval state or in the prompt, not in another output-remediation lever. Enforcement-only A/B: each governed config differs from its baseline by exactly two lines (run, callable). One systematize/v0001 and one test_set/v0001 shared by every run, so no stage was regenerated. Provenance: agent_guarded.py was written 11:35:19, the shipped run started 11:36:54 and ended 12:20:12, and the file has not been touched since. The intermediate configs eval_config.governed.yaml and eval_config.governed_grounded.yaml are retained because summary.md cites their numbers as the progression, but the committed agent_guarded.py is the grounded+scoped code and will NOT reproduce those two runs. Only eval_config.governed_grounded_v2.yaml is reproducible from this tree. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * fix(library): use FutureWarning for the moved-scenario shim, per Yeming's review DeprecationWarning is suppressed by Python's default warning filters outside pytest/-W. The shim's whole purpose is to tell config authors their behavior:{preset: travel_planner}-style config has been reclassified without breaking it -- with DeprecationWarning, that notice was invisible to anyone running assert-ai run directly, only visible under pytest (which re-enables DeprecationWarning by default). FutureWarning is shown by default in normal script execution, which is the actual audience for this warning. * feat(example): travel_langgraph_planner ran through SKILL workflow. * chore(examples): strip ACS artifacts from the 8 worked domains Keep only the baseline agent, the baseline eval configs, the Clarity Protocol design record and the README in each domain. Deleted (57 files): acs/ manifests + rego policy .......... 30 agent_guarded.py ...................... 8 eval_config.governed*.yaml ............ 18 prompt_agents/verify_gates.py ......... 1 verify_gates.py goes with them: it imports agent_guarded and loads acs/, so it cannot function once those are gone. Kept deliberately, though a literal "delete what the skill made" would have removed them: - __init__.py in billing_support_agent, career_health_assessment and prompt_agents. These were added by the skill, but are required for examples.<domain>.agent to import. - mock_tools.py, mcp_tools.py and docs/ (azure_doc_qa); tools.py (change_control_agent, science_research_agent); the five health_assistant*.yaml specs (prompt_agents). All are referenced at import time or read at runtime. All 17 remaining baseline eval_config.yaml target agent.py and contain no acs/ or guarded references, so the baseline side of each A/B is intact. Clarity Protocol is left as-is. 8 of its 205 files still describe acs/ and agent_guarded.py; that is prose, not imports, and is correct for a historical design record. Verified: all 8 examples.<domain>.agent modules import cleanly. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * feat(examples): rewrite the 8 domain READMEs for the stripped branch. * feat(examples): aligned READMEs to impermissible/permissible behavior violation wording. * feat(readme): update readme with SKILL get started. * docs(examples): align incident_triage_agent row with main's baseline-only layout Merging origin/main brought in 054797f, which reduced incident_triage_agent to a baseline-only, one-behavior-per-YAML example and deleted eval_config_naive_prompt.yaml, eval_config_guarded.yaml, eval_config_guarded_gepa.yaml, and incident-triage.guardrails.yaml. Git merged cleanly because no file was touched on both sides, but the top-level examples/README.md row still advertised all four configs and the ACS + GEPA 4-variant matrix -- stale on main as well, since 054797f never updated this table. Repoint the row at what the example actually ships: behaviors/ as the recommended one-behavior-per-YAML split, with eval_config_baseline.yaml as the bundled overview. Drop the ACS/GEPA framing (that demo is superseded by #262 and its guardrails file is deleted); ACS remains covered by the acs_guardrails row. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * docs: repoint example config paths at the evals/<risk>/ layout This branch moved each example's eval_config.yaml under evals/<risk>/ so one config probes one risk, but docs across the repo still pointed at the old top-level paths. Every 'assert-ai run --config examples/travel_planner_langgraph/eval_config.yaml' (and two 'assert-ai init --from' variants) referenced a file this branch deleted, so the copy-paste quickstarts in AGENTS.md, docs/getting-started.md, and docs/guides/securing-agents-with-acs.md all failed. Repoint those at evals/budget-overrun/eval_config.yaml, matching the root README. In examples/README.md also fix the science_research_agent row and correct the canonical example's trace backend, which the config now sets to otel rather than phoenix. azure_doc_qa/IMPROVEMENT_JOURNEY.md is left pointing at its original bundled config on purpose: it is a historical log whose rates came from a single run scoring 9 judge dimensions over 56 test cases, so repointing it at a single-risk config would misattribute those numbers. Add a note recording that the config was since split, with links to the replacements. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b * fix(examples): close remaining #296 review gaps - Add examples/benchmark/README.md, the deliverable Ahmed explicitly asked for and accepted ("i think yes, adding the readme would be more clear"). Explains this is a throughput-scale variant of the flagship travel_planner_langgraph example -- same target, same explicit_constraint_violation_failures preset already used by behaviors/constraints.yaml, deliberately non-adversarial context: -- not a new agent or behavior. - Register examples/benchmark/ in examples/README.md's selection table and layout tree so it is actually discoverable, matching this PR series' own stated goal. - Fix the one sibling config eval_config.yaml itself missed in the prior YAML-style pass: the overrefusal rubric was still the hard-wrapped single-quoted scalar form; now literal-block style like every other rubric/context field in this example. Verified byte-for-byte semantic equivalence via yaml.safe_load diff -- pure style fix. 51/51 presets clean, 89/89 targeted tests pass. * feat(example): added the rest of eval_config.yaml for career_health_assessment. * docs(examples): commit the behavior taxonomy alongside eval_config. * fix(tests): restore-judge preset coverage lost to a silent skip. * fix(tests): collect and trigger the skill's test suite. * fix(deps): bound arize-phoenix below the release that breaks the CI. * feat(example): give billing_support_agent a real policy instead of a prompt. * fix: close the four non-blocking PR review follow-ups. * fix: close the second-round PR review items. * fix(examples): keep evals atomic and runnable Remove generated discovery artifacts and bundled configs, narrow composite behaviors, and align example navigation and commands with the runnable suites. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * chore(examples): defer travel eval layout to stacked cleanup Remove the overlapping travel-planner behavior configs so the downstream examples PR owns the canonical flat evals layout. Keep the atomic benchmark update, but make its documentation independent of the removed path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7cb46daf-b5ce-4ad5-a85d-977737e5c02b * fix(ci): cap Phoenix for Python 3.11 Phoenix 19.18+ crashes while pytest auto-loads its plugin on Python 3.11. Keep the existing compatible lock resolution, constrain the optional dependency, and make dependency metadata changes trigger regression CI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7cb46daf-b5ce-4ad5-a85d-977737e5c02b * fix(skill): align release metrics and example curation * docs: normalize getting-started line endings * fix(examples): restore the career health worked domain * fix(examples): remove career health worked domain Restore the intended six-domain curated example surface and remove the navigation entries reintroduced in 5764e0d. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs(examples): remove career health navigation Keep the root and examples indexes aligned with the intended six worked domains. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Alex Ngo <t-alexngo@microsoft.com> Co-authored-by: changliu2 <changliu2@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Jake Present <jakepresent1@gmail.com> Copilot-Session: 7cb46daf-b5ce-4ad5-a85d-977737e5c02b Copilot-Session: eb38db79-6f6e-4f57-9e57-c7496498048b

github-advanced-security AI found potential problems Jul 9, 2026

View reviewed changes

Chang Liu (changliu2) and others added 3 commits July 10, 2026 15:39

Chang Liu (changliu2) marked this pull request as ready for review July 14, 2026 19:49

Chang Liu (changliu2) requested review from Aaron Aspinwall (AaronAspinwall123), Jake Present (jakepresent), Minsoo Thigpen (minthigpen) and tangym as code owners July 14, 2026 19:49

Chang Liu (changliu2) mentioned this pull request Jul 22, 2026

feat(cli): results matrix — behavior × arm comparison table #274

Open

Chang Liu (changliu2) and others added 4 commits July 23, 2026 14:01

Merge remote-tracking branch 'origin/main' into changliu2/bank-1behav…

41b910c

…ior-configs

Chang Liu (changliu2) mentioned this pull request Jul 23, 2026

fix(examples): incident_triage_agent — baseline-only, one-behavior-per-YAML (ACS demo superseded by #262) #271

Merged

Chang Liu (changliu2) and others added 2 commits July 24, 2026 15:04

Merge remote-tracking branch 'origin/main' into changliu2/bank-1behav…

1e8efd1

…ior-configs

Chang Liu (changliu2) and others added 3 commits July 24, 2026 15:48

docs: fix bank_manager_agent_control README to point to PDF deck, not…

a595285

… nonexistent index.html Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Aaron Aspinwall (AaronAspinwall123) approved these changes Jul 28, 2026

View reviewed changes

Chang Liu (changliu2) merged commit a6e34b5 into main Jul 28, 2026
4 checks passed

Navigation Menu

Uh oh!

docs(examples): bank-manager agent-control demo (AIEWF) + talks slideshow - #262

docs(examples): bank-manager agent-control demo (AIEWF) + talks slideshow#262
Chang Liu (changliu2) merged 16 commits into
mainfrom
changliu2/aiewf-bank-demo

Chang Liu (changliu2) commented Jul 9, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Chang Liu (changliu2) commented Jul 24, 2026

Uh oh!

Aaron Aspinwall (AaronAspinwall123) commented Jul 27, 2026

Uh oh!

Chang Liu (changliu2) commented Jul 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Chang Liu (changliu2) commented Jul 9, 2026

What

Scope / decisions (please sanity-check)

Verification

Draft — for review, not merge

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Chang Liu (changliu2) commented Jul 24, 2026

Maintainer review — do the ACS mitigations make sense?

What holds up ✅

Hardening notes (fine for a demo; documented in the new README section)

Uh oh!

Aaron Aspinwall (AaronAspinwall123) commented Jul 27, 2026

Uh oh!

Chang Liu (changliu2) commented Jul 28, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants