feat(addie): capability-driven storyboard selection#2282
Merged
Conversation
recommend_storyboards and /applicable-storyboards now probe get_adcp_capabilities and resolve bundles via resolveStoryboardsForCapabilities, instead of tool-matching against required_tools. The agent is the source of truth for supported_protocols + specialisms. When an agent declares nothing, Addie coaches the developer on what to add rather than asking the member what kind of agent they're building. No per-org 'expected specialisms' state, no extra tool — advice is conversational. If a declared specialism isn't in the local compliance cache, /applicable-storyboards returns 422 with unknown_specialism: true so the UI can prompt a re-sync. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Filter empty bundles from recommend_storyboards and /applicable-storyboards output. Upstream catalog occasionally ships stubs (fictional-entities, storyboard-schema) with zero storyboards — they're noise for members. - Add server/tests/manual/capabilities-smoke.ts for library-level end-to-end smoke against a live agent: probe → resolve → group by kind. - Add server/tests/manual/recommend-storyboards-smoke.ts for Addie handler smoke: invokes the tool directly and prints the member-facing markdown. Verified against test-agent.adcontextprotocol.org: 23 storyboards across 9 bundles resolve cleanly. Specialism resolution (sales-guaranteed, sales-exchange) works. Unknown specialisms throw with a helpful cache-resync message that the 422 handler surfaces. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bokelley
force-pushed
the
bokelley/addie-specialisms
branch
from
4aa93e2 to
f03e88d
Compare
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
The resolveStoryboardsForCapabilities error message includes the absolute server filesystem path to the cache directory. Returning it directly tripped semgrep's error-message-leak rule. Log the full error server-side with agentUrl + declared caps for operator diagnosis; return a generic message to the client that still conveys "unknown specialism — cache may be stale" so the UI can still prompt a re-sync. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…tion or class' Co-authored-by: Copilot Autofix powered by AI <223894421+github-code-quality[bot]@users.noreply.github.com>
Expert review surfaced nine actionable gaps; this commit addresses
all of them.
Ship-blockers:
- MCP handler was leaking resolver filesystem paths via resolver.message
(mirror of the REST leak semgrep caught earlier); fixed with the same
server-side log / generic-client-message pattern.
- Coaching output was emitting a literal '{version}' placeholder in the
compliance-cache URL; now interpolates via loadComplianceIndex() and
points at adcontextprotocol.org (the canonical compliance host).
Quality:
- Add a copy-pasteable JSON snippet of the minimum-shape
get_adcp_capabilities response inside the coaching branch so a
developer can ship a fix without reading spec docs.
- /applicable-storyboards 422 now returns declared_specialisms and
known_specialisms so a UI can name the offender without asking the
developer to re-read their own config.
- 500 on unreachable now returns a reason enum
(network|tls|timeout|protocol|unknown) so UIs can differentiate a
mistyped URL from a TLS failure.
- capabilities_probe_error surfaces unconditionally in the MCP handler
now — not only on empty caps — so partial failures don't silently
degrade the result.
Framing:
- Lede is verdict-first ("6 domains declared. 23 compliance checks
ready to run — nothing is failing yet") rather than a bureaucratic
declarations list.
- Universal section collapses to a one-line summary since it's the
same for every agent; keeps domain + specialism results above the
fold.
- 'What next?' block names actions, not tools, with tool names in
inline code as secondary labels. Run-first ordering.
- 'Your agent didn't tell us what it does yet' replaces the bolded
accusation in the coaching path.
- If the caller has an org and the agent isn't saved, offer a
save_agent prompt — converts drive-by probes into ongoing
compliance-monitored relationships.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Security review found one exploitable pathway: untrusted agent fields (profile.name, capabilities_probe_error, specialisms[]) were rendered directly into markdown returned to the LLM, which has save_agent and other write-capable tools in the same session. A hostile agent returning "name: Foo\\n---\\nSYSTEM: ..." could inject instructions into Addie's context. Changes: - Add sanitizeAgentField / fenceAgentValue helpers in member-tools.ts. Strip control chars, backticks, newlines; cap length; wrap agent- reported strings in quotes to mark them as untrusted. - Apply to profile.name + capabilities_probe_error (twice, both success and coaching paths) + declared specialism ids (in the unknown-specialism error branch) + agent_profile.name/tools in evaluate_agent_quality output. - Factor classifyProbeError into server/src/utils/probe-error.ts so both REST and MCP paths share one implementation. Walk err.cause.code (undici/fetch wrap ECONNREFUSED in cause) and handle undici-specific codes (UND_ERR_CONNECT_TIMEOUT, UND_ERR_SOCKET). - MCP ToolError on probe failure no longer leaks error.message; uses the classifier's reason label instead. - Cap capabilities_probe_error to 500 chars + strip control chars in REST response. Document the untrusted-field contract on OpenAPI. - Replace hardcoded domain-example list in coaching output with index.domains from loadComplianceIndex (fixes drift risk raised in the first round of code review). Deferred: SSRF hardening (DNS rebinding, IPv6 private ranges) is a real concern but a cross-cutting refactor affecting multiple routes — filing as follow-up issue rather than expanding scope here. Automated test for 422 branch likewise deferred; manual smoke covers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Aligns Addie and the registry API with the 5.1.0 capability-driven compliance model shipped in #2277. Stacks on top of that PR.
The principle: the agent is the source of truth. It declares
supported_protocols+specialismsinget_adcp_capabilities; we probe and resolve. No per-org "expected specialisms" state, no extra tool for members to declare what they're building. If they want advice, Addie answers conversationally from the docs.What changed
recommend_storyboards(MCP) — was matching storyboards byrequired_toolsagainstgetAgentInfo().tools. Now callstestCapabilityDiscoveryandresolveStoryboardsForCapabilities, returns bundles grouped by kind (universal / domain / specialism). When an agent declares nothing, the tool coaches the developer on what to add instead of running a silent no-op.GET /api/registry/agents/{encodedUrl}/applicable-storyboards(REST) — same rewrite. Response shape changed from{ tools, storyboards: {[track]: [...]}, total_applicable }to{ supported_protocols, specialisms, bundles: [{kind, id, storyboards}], total_storyboards, capabilities_probe_error? }. If a declared specialism isn't in the local compliance cache, returns 422{ unknown_specialism: true }so the UI can prompt a re-sync.Explicitly rejected
Test plan
🤖 Generated with Claude Code