Skip to content

fix(addie): grade_agent_signing auto-skips capability-profile mismatches#3452

Merged
bokelley merged 2 commits into
mainfrom
bokelley/grader-capability-probe
Apr 28, 2026
Merged

fix(addie): grade_agent_signing auto-skips capability-profile mismatches#3452
bokelley merged 2 commits into
mainfrom
bokelley/grader-capability-probe

Conversation

@bokelley

Copy link
Copy Markdown
Contributor

Summary

When the first end-to-end grade demo ran in prod Slack, Addie reported 31 pass / 2 fail / 6 skip against https://test-agent.adcontextprotocol.org/mcp-strict and diagnosed the 2 fails as real verifier bugs. They aren't:

  • Vector 007-missing-content-digest tests that an agent declaring covers_content_digest: 'required' rejects digest-less signatures.
  • Vector 018-digest-covered-when-forbidden tests that an agent declaring covers_content_digest: 'forbidden' rejects digest-covering signatures.

/mcp-strict declares covers_content_digest: 'either', so both vectors are testing modes the agent didn't promise to enforce. The in-process gradeRequestSigning() has an agentCapability option that auto-skips these as capability_profile_mismatch; the CLI surface doesn't expose it.

This wrapper now does the equivalent CLI-side:

  • Anonymously probes get_adcp_capabilities for request_signing.covers_content_digest.
  • Maps mode → inapplicable vectors and passes --skip <ids> to the CLI.
  • Exposes a content_digest_mode parameter for routes where the anonymous probe is auth-gated.
  • Probe-failure path preserves today's no-skip behavior — better to over-report than silently swallow a real verifier bug.

Result

Against https://test-agent.adcontextprotocol.org/mcp-strict:

passed failed skipped
Before 31 2 (false) 6
After (mode='either') 31 0 8

The 2 extra skips are correctly classified as capability-profile mismatches.

Follow-ups

  • The in-process gradeRequestSigning() does richer capability-aware skipping than just content-digest. Long-term the right fix is to upstream a --auto-skip-capability-mismatches flag to @adcp/client's CLI so this wrapper doesn't need to maintain a hardcoded vector→mode map.
  • The probe is anonymous today. For auth-gated routes (test agent's strict variants) it falls back to no-skip unless the caller passes content_digest_mode. Could plumb evaluate_agent_quality's resolveAgentAuth here so saved credentials drive the probe automatically.

Test plan

  • npx vitest run tests/unit/auth-grader-tools.test.ts — 10 tests pass (was 6; new contentDigestSkipsForMode cases for null / either / required / forbidden).
  • Server typecheck clean.
  • CLI grade against /mcp-strict with --skip 007-...,018-... produces 31 pass / 0 fail / 8 skip against prod test agent.
  • After deploy: re-run the Slack demo with content_digest_mode: either and confirm clean report.

🤖 Generated with Claude Code

bokelley and others added 2 commits April 28, 2026 11:57
After the wrapper started running grades end-to-end (#3424), Addie's
first prod demo against /mcp-strict reported "31 pass / 2 fail" — and
diagnosed the 2 fails as real verifier bugs. They aren't: vectors 007
and 018 test required-mode and forbidden-mode content-digest
enforcement respectively, but /mcp-strict declares
covers_content_digest: 'either', so neither vector applies. The
in-process gradeRequestSigning() has an `agentCapability` option that
auto-skips these as capability_profile_mismatch; the CLI surface
doesn't expose it.

This wrapper now does the equivalent CLI-side:
- Anonymously probes get_adcp_capabilities for
  request_signing.covers_content_digest.
- Maps the declared mode → vectors that don't apply (007 when 'either'
  or 'forbidden', 018 when 'either' or 'required').
- Passes --skip <ids> to the CLI for the matching vectors.
- Caller can short-circuit with content_digest_mode for auth-gated
  routes where the anonymous probe fails. Probe-failure path
  preserves today's no-skip behavior — better to over-report than
  silently swallow a real verifier bug.

Validated against test-agent.adcontextprotocol.org/mcp-strict:
- Without skip: 31 pass / 2 fail / 6 skip (vectors 007, 018 false-fail).
- With auto-skip: 31 pass / 0 fail / 8 skip (clean report; probe
  succeeds when called with content_digest_mode=either).

Real long-term fix is upstreaming a capability-aware --auto-skip flag
to @adcp/client's CLI; the wrapper drops the hardcoded mapping then.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Security review on PR #3452 (🟡): the bare fetch() in
probeContentDigestMode bypassed the SSRF gate that the grader CLI's
ssrfSafeFetch enforces. A caller with an authenticated session could
supply a URL whose DNS A record resolves to 127.0.0.1 / 169.254.169.254 /
RFC1918 — fetch would hit the loopback/metadata endpoint before the
grader CLI's own SSRF gate could refuse, and an error response would
land in logger.debug.

Plumb the local validateFetchUrl from server/src/utils/url-security.ts
which DNS-resolves the hostname and rejects any address in private,
link-local, loopback, CGNAT, IPv6 ULA, or always-blocked ranges. Add
a 64 KiB body cap matching ssrfSafeFetch's default. Drop the agent URL
from the probe-failure debug log to close the info-disclosure side
channel. Set redirect: 'manual' so a 30x to a private host can't move
the connection laterally.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@bokelley bokelley merged commit 1cf2b16 into main Apr 28, 2026
14 checks passed
@bokelley bokelley deleted the bokelley/grader-capability-probe branch April 28, 2026 16:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant