docs: validate adapter agents with mock upstream fixtures#3826
Conversation
…oryboard + traffic
Makes the three validation gates we just used by hand into a CI test that
runs on every commit:
1. `npx tsc` with --strict + noUncheckedIndexedAccess +
exactOptionalPropertyTypes + noPropertyAccessFromIndexSignature +
2 other hardening flags. Catches latent type errors that default
`--strict` misses.
2. Boot the published mock-server signal-marketplace as upstream,
boot the example agent, run `adcp storyboard run signal_marketplace`,
assert `summary.steps_failed === 0`. Catches AdCP wire-shape
regressions.
3. After the storyboard run, GET /_debug/traffic and assert each
of `_lookup/operator`, `/v2/cohorts`, `/v2/activations` was hit
≥1 time. Catches façade regressions where the adapter returns
shape-valid responses without exercising upstream.
Why these three together: tsc catches type errors at build time;
storyboard catches wire-shape errors when exercised; traffic catches
shape-valid-but-fake responses. Each is a distinct failure mode the
others miss.
Adversarial validation: sabotaged the example (replaced the upstream
listCohorts call with `[]`) — gates 2 and 3 fail with distinct,
actionable messages ("storyboard reported N failed steps" + "These
upstream routes had zero hits — the adapter is a façade for them").
Gate 1 still passes because the regression is runtime-only, which is
correct: the three gates cover orthogonal failure modes.
CI wall: 18s (tsc 6.3s + storyboard 3.7s + traffic 0.04s + boot/teardown).
Acceptable for a per-commit gate.
Wired into ci.yml after `npm test`. Separate `test:examples` script
keeps it isolatable for local iteration. Pretest:examples runs
`npm run build` so the dist/ artifacts are present.
This is the pattern the validate-with-mock-fixtures spec guide
(adcontextprotocol/adcp#3826) describes — applied here to our own
reference adapter so what we tell adopters to do is what we
ourselves do. Future hello_seller_adapter_<specialism> additions
each get their own test/examples/*.test.js with the same shape.
…ference (#1274) * docs(examples): hello_seller_adapter_signal_marketplace — worked passing reference A complete v6 typed-platform AdCP signals adapter that wraps an upstream signal-marketplace platform via HTTP. Demonstrates the full adapter pattern: account.operator → upstream X-Operator-Id resolution, get_signals → /v2/cohorts, activate_signal → /v2/cohorts/{id} + /v2/destinations + POST /v2/activations, with both `platform` and `agent` destination branches and proper schema-conforming response shapes. Designed as the starting point adopters fork. SWAP markers throughout mark every place the example assumes the published mock as upstream; adopters replace those with calls to their real backend's HTTP/SDK client. The AdCP-facing platform methods stay identical. ## Why this exists Every seller agent is an adapter — even ones whose "upstream" is internal infrastructure are translating AdCP wire calls into backend calls. Reading the spec + skill and synthesizing a correct agent from scratch is hard; empirically (from blind LLM-built adapters in our SDK dogfood) most first-pass attempts had at least one missing field, missing error code, or misshapen response that cascaded through the storyboard. Giving adopters a working starting point removes most of that friction. ## Validated end-to-end Against the published `adcp mock-server signal-marketplace`: steps_passed: 7 steps_failed: 0 steps_skipped: 4 (all missing_tool — tools this specialism doesn't claim) Traffic counters confirm real upstream integration: GET /_lookup/operator: 6 GET /v2/cohorts: 4 GET /v2/cohorts/{id}: 2 GET /v2/destinations: 2 POST /v2/activations: 2 This is the first clean traffic-verified pass on any specialism in the matrix-blind-fixtures lineage. Documents what "this is what a clean E2E pass looks like" actually means. ## Spec/schema gotchas surfaced while building Each documented inline so the next reader doesn't re-discover them: - `signal_ids` is `signal_id[]` (array of provenance objects), NOT array of strings — schema says so explicitly. Comparing against bare strings silently returns no matches. - `ActivationKey` is a oneOf: `type: 'key_value'` puts `key` and `value` at the TOP level of the activation_key object, NOT nested under a `key_value` sub-field. `value` MUST be string. - The mock's `agent_activation_key` is `{ agent_segment: <string> }` (an object), not a bare string — extract `.agent_segment` for the ActivationKey.value field. - `signal_spec` is a free-text semantic brief; passing it to the upstream's substring-match `q` filter loses real-platform-style semantic matching and breaks against literal-text mock filtering. Pass it to a real semantic index in production; for the mock, fetch the full catalog and filter client-side via signal_ids. ## Run # Demo: boot mock + agent npx @adcp/sdk@latest mock-server signal-marketplace --port 4150 UPSTREAM_URL=http://127.0.0.1:4150 \ npx tsx examples/hello_seller_adapter_signal_marketplace.ts # Validate adcp storyboard run http://127.0.0.1:3001/mcp signal_marketplace \ --auth sk_harness_do_not_use_in_prod curl http://127.0.0.1:4150/_debug/traffic ## Refs - Linked from the validate-with-mock-fixtures spec guide (adcontextprotocol/adcp#3826) - Companion to skills/build-signals-agent/ (skill should reference this example as the starting point for adapter-style agents) * fix(example): tighten hello signals adapter to pass strictest typecheck After review on PR #1274 the example was 435 lines and failed under `--strict --noUncheckedIndexedAccess --exactOptionalPropertyTypes --noPropertyAccessFromIndexSignature`. Eight unique errors including two real bugs (Account.authInfo missing, capabilities missing creative_agents/channels/pricingModels). The example "passed" only because the storyboard never exercised the missing branches. Cleanup: * Add `authInfo: ctx?.authInfo ?? { principal: 'anonymous' }` to accounts.resolve return — the field is required on Account and the framework strips it before emitting on the wire (typed intentionally so adapters thread the principal through to resource handlers, but signals-only adapters that don't authorize against it still need to pass it). * Declare `creative_agents: [] as const, channels: [] as const, pricingModels: ['cpm'] as const` on capabilities. Empty arrays are correct for signals-only platforms (don't sell media, don't compose with creative agents). Filed adcp-client#1278 for the underlying DX issue: these required fields are surprising for signals-only platforms; should be optional or specialism- conditional. * Narrow signal_ids filter on the `source: 'catalog'` discriminator before reading `data_provider_domain`. SignalID is a oneOf — catalog vs agent — and only catalog has provenance. * `process.env['FOO']` instead of `.FOO` (TS4111). * Cast ctx.account in resolveSessionKey since framework types it as broadly typed there. * Build RequestInit with conditional body-set instead of explicit `body: undefined` (exactOptionalPropertyTypes). * Extract httpJson<T>(method, path, opts) helper — collapses 5 UpstreamClient methods into one generic + 5 typed entry points. Each entry point still annotated with `// SWAP:` so adopters see exactly what to replace when wiring to a real backend. * Tighten comments — keep spec-gotcha learnings inline (the four "things schema validators catch but type checkers don't"), drop paragraph-length prose where one line works. Result: 360 lines (was 435), 0 errors under the strictest realistic adopter config (--strict + 4 extra flags), runtime gates unchanged (7/11 passed, 0 failed, 4 skipped — all `missing_tool`; traffic counters verify all 5 upstream routes hit). The two real bugs (authInfo + capabilities required fields) are now documented inline so adopters who fork this skip both. The discriminated-union narrowing is also documented inline. Each is the kind of friction that would cascade-fail an adopter's first run; the example now sidesteps all of them. * test(ci): three-gate test for hello signals adapter — strict tsc + storyboard + traffic Makes the three validation gates we just used by hand into a CI test that runs on every commit: 1. `npx tsc` with --strict + noUncheckedIndexedAccess + exactOptionalPropertyTypes + noPropertyAccessFromIndexSignature + 2 other hardening flags. Catches latent type errors that default `--strict` misses. 2. Boot the published mock-server signal-marketplace as upstream, boot the example agent, run `adcp storyboard run signal_marketplace`, assert `summary.steps_failed === 0`. Catches AdCP wire-shape regressions. 3. After the storyboard run, GET /_debug/traffic and assert each of `_lookup/operator`, `/v2/cohorts`, `/v2/activations` was hit ≥1 time. Catches façade regressions where the adapter returns shape-valid responses without exercising upstream. Why these three together: tsc catches type errors at build time; storyboard catches wire-shape errors when exercised; traffic catches shape-valid-but-fake responses. Each is a distinct failure mode the others miss. Adversarial validation: sabotaged the example (replaced the upstream listCohorts call with `[]`) — gates 2 and 3 fail with distinct, actionable messages ("storyboard reported N failed steps" + "These upstream routes had zero hits — the adapter is a façade for them"). Gate 1 still passes because the regression is runtime-only, which is correct: the three gates cover orthogonal failure modes. CI wall: 18s (tsc 6.3s + storyboard 3.7s + traffic 0.04s + boot/teardown). Acceptable for a per-commit gate. Wired into ci.yml after `npm test`. Separate `test:examples` script keeps it isolatable for local iteration. Pretest:examples runs `npm run build` so the dist/ artifacts are present. This is the pattern the validate-with-mock-fixtures spec guide (adcontextprotocol/adcp#3826) describes — applied here to our own reference adapter so what we tell adopters to do is what we ourselves do. Future hello_seller_adapter_<specialism> additions each get their own test/examples/*.test.js with the same shape. * refactor(examples): use new SDK ergonomics — drop optional empty arrays + authInfo + use public mock-server export
|
Pushed What changedConvergent block (3 reviewers): non-normative framingOriginal implied
Convergent block: language-agnostic claim vs Python-only sketchOriginal opened "language-agnostic" then proved it with ~40 lines of Python only. Two reviewers (docs + product) flagged the self-contradiction. Fixed:
Reviewer-specific issues addressed
Discoverability (docs expert)Two new inbound links added:
Tone
Open questions reviewers raised that need spec-side inputThese are reasonable to leave for spec-maintainer judgment rather than self-resolve:
Diff is ~100 lines smaller, ~50 lines added — the non-normative framing replaced repeated implications throughout the doc rather than adding decorations. Ready for spec-side review. |
Adds `docs/building/validate-with-mock-fixtures.mdx` — a four-step recipe adopters can follow to validate AdCP agents that wrap an upstream platform (DSP, SSP, retail data, creative server, signal marketplace), in any language: 1. Boot a published mock upstream for the specialism 2. Point your agent at the mock as its upstream 3. Run the storyboard runner for AdCP wire-contract conformance 4. Assert traffic counters at /_debug/traffic for façade resistance Storyboards alone validate wire shape but cannot detect façade adapters that return shape-valid AdCP responses without integrating with the upstream. The traffic counter gate closes that hole. Both signals matter; one without the other is incomplete. The recipe is language-agnostic. The reference TS SDK (@adcp/client) ships four mock specialisms that other-language SDKs can also implement; the mock upstreams expose plain HTTP, so the agent under test can be in any language. Honest about limitations: storyboards under-cover payload variety (#3785), cascade fragility silently skips downstream steps on early-step shape gaps (#3796), mock seeds may not match storyboard fixture inputs, and the traffic gate is necessary but not sufficient (sophisticated façades with synthetic placeholder data still pass). Linked from validate-your-agent.mdx via a single Info callout in the preamble so adapter authors find it without needing to know the new file exists.
…/webhook caveats Three-way reviewer feedback (docs-expert, ad-tech-protocol-expert, adtech-product-expert): 1. Make explicit non-normative + harness-convention framing (top <Note>) — the original implied that /_lookup and /_debug/traffic are spec-level. Now framed as @adcp/client reference implementation; alternatives may diverge. 2. Reframe as 'pre-staging gate' (not 'validation/replacement'). Recipe complements storyboards + staging integration, doesn't replace either. 3. Drop Python-only sketch. The original violated its own 'language-agnostic' claim. Replaced with a sidecar pattern note: agent stays in its language, TS CLI runs as service container. 4. Bare 'adcp' → 'npx @adcp/client@latest'. Matches existing adcp docs convention. 5. Tone: removed 'dogfood' + 'empirically observed' phrasings. Now docs voice. 6. Added concrete GitHub Actions YAML example with the traffic-counter assertion. 7. Added threshold guidance (= 1 per route is the right floor; richer assertions are encoded in storyboard payloads, not the gate). 8. Mock specialism table: added caveat that AdCP-side identifier names (account.advertiser/operator/publisher) are SDK conventions for tenant binding, not normative AdCP terms. Plus a note that the four specialisms are not exhaustive. 9. Limitations section restructured into three sub-headings: - Storyboard limitations (#3785 payload variety) - Runner/tooling caveats (#3796 cascade — moved here, was wrongly tagged as spec issue) - Traffic gate limitations — added two from protocol-expert review: * idempotency replay isn't exercised (façade that doubles writes passes hit count) * outbound webhook delivery isn't on upstream traffic counters 10. Added inbound links from build-an-agent.mdx (so adapter-shape decisions trigger the discovery) and compliance-catalog.mdx (so adopters reading the storyboard taxonomy see the complementary gate). Reviewers' concerns about certification implications — addressed by the top <Note> explicitly disclaiming compliance-tier status.
…at new paths PR #4022 (IA Phase 2) relocated the linked pages after this branch diverged. Rebase silently dropped the three discoverability callouts because their source paths no longer existed. Re-author at the new paths: - Move docs/building/validate-with-mock-fixtures.mdx into docs/building/verification/ alongside the other validation pages. - Add discovery sentence to docs/building/by-layer/L4/build-an-agent.mdx. - Add Tip block to docs/building/verification/compliance-catalog.mdx. - Add Info callout to docs/building/verification/validate-your-agent.mdx. - Update internal links inside the moved doc to canonical post-IA paths. - Register the page in docs.json nav (both top-level and L4 builders nav). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
47b5486 to
86f2601
Compare
The Info callout added to dist/docs/3.0.7/building/validate-your-agent.mdx in #3826 linked to /docs/building/validate-with-mock-fixtures, but the page actually lives at /docs/building/verification/validate-with-mock-fixtures (the IA Phase 2 path that #3826 itself migrated to in the live docs/ copy). The link in dist/ was missed during that migration, fails CI's broken-links check on every PR that touches docs/ or addie source code, and is currently red on origin/main. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous fix(docs) commit tried to correct the link path. But the target page (validate-with-mock-fixtures) doesn't exist in the dist/docs/3.0.7/ snapshot at all — it was added to the live docs/ tree after 3.0.7 was cut. Either spelling of the link breaks one of two checks: broken-links (target missing) or check-dist-links (unversioned link in a versioned snapshot). The 3.0.7 snapshot should be immutable; PR #3826 violated that by adding the Info callout pointing at content that didn't exist when 3.0.7 shipped. Restore the snapshot to its pre-3826 state. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…truth (#4500) * feat(agents): rescue long advisors via -deep suffix, unify source of truth `.agents/roles/` is now the single source for all subagent prompts, with two forms per role: - `{name}.md` — short triage checker (terse, PR-bound, used by triage routine) - `{name}-deep.md` — long-form design advisor (full reasoning) for open-ended work like MCP tool surface review or threat modeling Nine `-deep` counterparts are now reachable that had been silently shadowed by the short `.claude/agents/` versions for ~7 months: MCP agentic-API design review, context-window efficiency lens, threat modeling depth, curriculum architecture, etc. `.claude/agents/` and `.codex/agents/` are fully generated by `scripts/import-claude-agents.mjs`. The script gained a filename-matches-frontmatter-name check and a duplicate-name guard. Addie's expert-panel system-prompt section filters out `-deep` variants so each persona appears once (server/src/addie/rules/index.ts), with a regression test. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(agents): address expert review on PR #4500 * fix(docs): correct broken-link in 3.0.7 snapshot validate-your-agent.mdx The Info callout added to dist/docs/3.0.7/building/validate-your-agent.mdx in #3826 linked to /docs/building/validate-with-mock-fixtures, but the page actually lives at /docs/building/verification/validate-with-mock-fixtures (the IA Phase 2 path that #3826 itself migrated to in the live docs/ copy). The link in dist/ was missed during that migration, fails CI's broken-links check on every PR that touches docs/ or addie source code, and is currently red on origin/main. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * chore: align schema registry adcp_version with package.json The legacy adcp_version alias in static/schemas/source/index.json had drifted to 3.0.12 (a non-existent release tag) while published_version and package.json are at 3.0.3. The pre-push verify-version-sync hook caught this; origin/main is in the same state but CI doesn't run the check, so it went unnoticed. Regenerated via `node scripts/build-schemas.cjs`; the only change is the one-line legacy alias. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(docs): remove invalid Info block from 3.0.7 snapshot The previous fix(docs) commit tried to correct the link path. But the target page (validate-with-mock-fixtures) doesn't exist in the dist/docs/3.0.7/ snapshot at all — it was added to the live docs/ tree after 3.0.7 was cut. Either spelling of the link breaks one of two checks: broken-links (target missing) or check-dist-links (unversioned link in a versioned snapshot). The 3.0.7 snapshot should be immutable; PR #3826 violated that by adding the Info callout pointing at content that didn't exist when 3.0.7 shipped. Restore the snapshot to its pre-3826 state. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Summary
Adds
docs/building/validate-with-mock-fixtures.mdx— a four-step recipe adopters can follow to validate AdCP agents that wrap an upstream platform (DSP, SSP, retail data warehouse, creative server, signal marketplace), in any language:Why this guide
Storyboards alone validate wire shape but cannot detect façade adapters — agents whose handlers return shape-valid AdCP responses without integrating with the upstream. Empirically observed (during SDK dogfood): an LLM-built adapter wrote OAuth client code, declared the import, and never called it from any handler. The handlers returned hardcoded shape-valid responses for every call. Storyboard graded `passing`. Real upstream traffic: zero requests.
Traffic counters close the hole. Both signals matter; one without the other is incomplete.
Why this lives in the spec repo
The pattern is language-agnostic:
The reference TS SDK (`@adcp/client`) ships four mock specialisms today; the recipe doesn't depend on the TS SDK specifically. Python, Go, Rust adopters can either point at the @adcp/client CLI for the mock-server step or implement equivalents in their language.
Honesty about limitations
Called out prominently rather than buried:
Discoverability
Linked from `validate-your-agent.mdx` via a single `` callout in the preamble. Adapter authors find it without needing to know the new file exists; the existing storyboard-only path remains intact for non-adapter agents.
Test plan
Refs
🤖 Generated with Claude Code