diff --git a/.changeset/implementer-dx-2607-2608.md b/.changeset/implementer-dx-2607-2608.md new file mode 100644 index 0000000000..807b08e86a --- /dev/null +++ b/.changeset/implementer-dx-2607-2608.md @@ -0,0 +1,14 @@ +--- +--- + +feat(docs+compliance): implementer DX — troubleshooting guide, known-ambiguities doc, idempotency probe fix (#2607, #2608) + +Adds two implementer-facing docs pages and an opt-out on the idempotency missing-key vector. + +`universal/idempotency.yaml` — the `create_media_buy_missing_key` step now declares `omit_idempotency_key: true`, which tells the runner to skip both its own `applyIdempotencyInvariant` and the SDK's auto-inject. Without the flag the reference `@adcp/client` SDK injects an `idempotency_key` before dispatch and the vector never probes the missing-key rejection path. The SDK already supports this opt-out (`Step.omit_idempotency_key`, wired through `skipIdempotencyAutoInject` in `task-map.js`); the missing piece was the storyboard declaration. `storyboard-schema.yaml` documents the field alongside the other step-level directives. + +`docs/building/implementation/storyboard-troubleshooting.mdx` — error pattern → root cause → fix reference. Sections cover unknown fixtures, RFC 9421 signature challenges, `errors[]` vs `adcp_error` envelope drift, context echo failures, capability-vector mismatches, `requireAuthenticatedOrSigned` composition, and `INVALID_STATE` vs `INVALID_TRANSITION` vocabulary. + +`docs/building/implementation/known-ambiguities.mdx` — open spec gaps with workarounds. Entries are removed as underlying issues close (#2389 past_start already closed; #2603/#2604/#2606/#2607 pending). Both pages cross-linked from `validate-your-agent.mdx` under a new "When a storyboard fails" section. + +docs.json nav registration covers both Implementation Patterns groups (duplicate arrays exist; both were updated). diff --git a/docs.json b/docs.json index 45adb9c057..8e98e70e03 100644 --- a/docs.json +++ b/docs.json @@ -144,7 +144,9 @@ "docs/building/implementation/a2a-response-extraction", "docs/building/implementation/security", "docs/building/implementation/webhook-verifier-tuning", - "docs/building/implementation/seller-integration" + "docs/building/implementation/seller-integration", + "docs/building/implementation/storyboard-troubleshooting", + "docs/building/implementation/known-ambiguities" ] }, { @@ -643,7 +645,9 @@ "docs/building/implementation/a2a-response-extraction", "docs/building/implementation/security", "docs/building/implementation/webhook-verifier-tuning", - "docs/building/implementation/seller-integration" + "docs/building/implementation/seller-integration", + "docs/building/implementation/storyboard-troubleshooting", + "docs/building/implementation/known-ambiguities" ] }, { diff --git a/docs/building/conformance.mdx b/docs/building/conformance.mdx index 922dcbdcb8..04515c1b72 100644 --- a/docs/building/conformance.mdx +++ b/docs/building/conformance.mdx @@ -129,6 +129,11 @@ Conformance is per-version; the suite is per-version. A 3.0-conformant agent is - **Grant or revoke certification.** The [AgenticAdvertising.org certification program](/docs/learning/overview) runs on top of this; conformance is necessary but not sufficient. - **Publish reference test vectors beyond those already in the suite.** The [Reference Test Vectors index](/docs/reference/test-vectors) catalogs the vector sets that ship today; broader task-level corpus lands incrementally between 3.0 GA and 3.1, scoped in [#2383](https://github.com/adcontextprotocol/adcp/issues/2383). +## When a storyboard fails + +- **[Storyboard troubleshooting](/docs/building/implementation/storyboard-troubleshooting)** — Error pattern → root cause → fix for the most common storyboard failures +- **[Known spec ambiguities](/docs/building/implementation/known-ambiguities)** — Open spec gaps with workarounds and issue links; entries are removed as underlying issues close + ## Further reading - **[Compliance Catalog](/docs/building/compliance-catalog)** — Full taxonomy of protocols and specialisms with storyboard IDs diff --git a/docs/building/implementation/known-ambiguities.mdx b/docs/building/implementation/known-ambiguities.mdx new file mode 100644 index 0000000000..66cf9963e9 --- /dev/null +++ b/docs/building/implementation/known-ambiguities.mdx @@ -0,0 +1,52 @@ +--- +title: Known spec ambiguities +description: "Open AdCP spec gaps that affect compliance testing — with workarounds and issue links. Entries are removed as the underlying issues close." +"og:title": "AdCP — Known spec ambiguities" +--- + +This page enumerates spec gaps that currently affect storyboard conformance. Each entry covers one of three patterns: a spec `MAY` branch where vectors assert one outcome, a response field storyboards assert that the schema does not yet require, or a testing-infrastructure quirk that prevents a vector from probing through the reference SDK. + +**Entries are removed as the underlying issue closes.** If an entry here doesn't match what you're seeing, check the GitHub issue for the latest state — the fix may have landed in a release you haven't pulled yet. + +## How to use this page + +If a storyboard fails on a behavior you believe is spec-conformant, search this page for the storyboard name or the assertion text. Each entry describes the gap, the workaround that gets you past the blocker, and the issue that tracks the fix. When the issue closes the entry here is removed — so always pair this page with the linked issue for the authoritative state. + +For the opposite direction — failures that are **not** in this list and do have clean fixes — see the [storyboard troubleshooting guide](/docs/building/implementation/storyboard-troubleshooting). + +## Current ambiguities + +### `check_governance` `conditions` field shape + +- **Schema**: `check-governance-response.json` defines `conditions[]` items as `{ field, required_value?, reason }` with `field` and `reason` required. The `status: conditions` status now requires `conditions` with `minItems: 1`. +- **Gap closed by**: [#2603](https://github.com/adcontextprotocol/adcp/issues/2603). The schema tightening lands in the protocol patch release following this entry's removal. +- **Workaround (until you pull the fix)**: emit `conditions[]` with the canonical `{ field, reason }` shape on every `status: conditions` response. Agents following the prose description already do this; the schema tightening just makes enforcement mechanical. + +### PRM required for non-OAuth agents + +- **Storyboard**: `universal/security.yaml` phases `oauth_discovery` + `mechanism_required`. +- **Gap**: the RFC 9728 / RFC 8414 probes run for every agent by default. API-key-only sandboxes were standing up fake issuer URLs to "pass" the probes, which is worse than skipping them. +- **Resolution**: [#2606](https://github.com/adcontextprotocol/adcp/issues/2606) — the storyboard narrative now explicitly directs API-key-only agents to declare `auth.api_key` in the test kit and omit PRM entirely. Optional-phase semantics make `oauth_discovery` failures non-fatal; `mechanism_required` passes via the API-key path. +- **Workaround**: if your agent has no OAuth issuer, do not serve `/.well-known/oauth-protected-resource/...`. Declare `auth.api_key` in your test-kit and let the OAuth discovery phase fail silently. + +### Idempotency missing-key probe via SDK + +- **Storyboard**: `universal/idempotency.yaml` step `missing_key/create_media_buy_missing_key`. +- **Gap**: the reference `@adcp/client` SDK auto-injects `idempotency_key` on mutating tasks, so a vector that tries to probe "missing key rejection" never reaches the agent with a missing key — the runner would inject one before dispatch. +- **Resolution**: [#2607](https://github.com/adcontextprotocol/adcp/issues/2607) — the step declares `omit_idempotency_key: true`, which signals the runner to skip both its own `applyIdempotencyInvariant` and the SDK's auto-inject. The request arrives at your agent without a key, letting the vector probe the rejection path honestly. +- **Workaround**: nothing required — honor the existing spec requirement (reject missing `idempotency_key` on mutating tasks with `INVALID_REQUEST` or `VALIDATION_ERROR`). + +### Response schema fields asserted by storyboards + +- **Storyboards**: `sales_catalog_driven` (catalog counts), `creative_ad_server` (pricing_options), `media_buy_seller/inventory_list_targeting` (property_list echo), `creative_ad_server` (vendor_cost required). +- **Gap**: historical drift between what storyboard vectors assert and what the response schemas require. +- **Resolution**: [#2604](https://github.com/adcontextprotocol/adcp/issues/2604). Audit complete: + - `sync-catalogs-response.json` now requires `item_count` on catalog entries when `action` is `created`/`updated`/`unchanged`. + - `property_list` / `collection_list` echo: already canonical via `packages[].targeting_overlay`. + - `list-creatives-response.json` `pricing_options`: already canonical (array, `minItems: 1`, items require `pricing_option_id`). + - `report-usage-request.json` `vendor_cost`: already required. +- **Workaround**: emit `item_count` on every non-failed/non-deleted catalog entry. Conformant agents already do this; the schema tightening catches gaps at `response_schema` validation. + +## When an ambiguity isn't listed + +If you're blocked on a behavior you believe the spec leaves ambiguous but it's not on this list, open an issue at [adcontextprotocol/adcp](https://github.com/adcontextprotocol/adcp/issues/new). Include the storyboard, the vector's assertion text, the conformant branch you picked, and why you believe the spec allows it. The fastest resolutions come from issues that cite the specific spec paragraph and the specific vector assertion — that's enough for a maintainer to either point you at an existing fix or confirm the gap and schedule it. diff --git a/docs/building/implementation/storyboard-troubleshooting.mdx b/docs/building/implementation/storyboard-troubleshooting.mdx new file mode 100644 index 0000000000..1dd599c7f6 --- /dev/null +++ b/docs/building/implementation/storyboard-troubleshooting.mdx @@ -0,0 +1,94 @@ +--- +title: Storyboard troubleshooting +description: "Common failure patterns when running AdCP compliance storyboards — missing fixtures, signature challenges, envelope drift, context echo, capability mismatches, and state-machine error codes." +"og:title": "AdCP — Storyboard troubleshooting" +--- + +When a compliance storyboard fails against your agent, the runner reports a step name and error text. This page maps the most common error patterns to their root causes and fixes, so you can resolve each class of failure without spelunking through SDK source or runner internals. + +Each section shows the error you'll see, what it means, and what to change in your agent. + +## Unknown fixture errors + +``` +× (unknown step): PRODUCT_NOT_FOUND: Package 0: Product not found: test-product +``` + +The storyboard's `sample_request` references a hardcoded ID (`test-product`, `test-pricing`, `campaign_hero_video`, `gov_acme_q2_2027`, etc.). The runner expects the agent to have that ID in its catalog before the mutating step runs. + +**Fix:** Implement `comply_test_controller` and honor the seed scenarios declared in the storyboard's `fixtures:` block. When `prerequisites.controller_seeding: true` is set, the runner auto-injects a fixtures phase that calls `seed_product`, `seed_pricing_option`, `seed_creative`, `seed_plan`, or `seed_media_buy` in foreign-key order before the main phases execute. + +See [Compliance test controller — Scenarios](/docs/building/implementation/comply-test-controller#scenarios) for the full seed contract. Agents that return `UNKNOWN_SCENARIO` on a seed call grade the storyboard `not_applicable` — they are not penalized for missing sandbox surface, but they cannot pass storyboards that depend on pre-seeded state. + +## Signature challenge missing on 401 + +``` +× (unknown step): expected error="request_signature_required", got error="(none)" +``` + +The storyboard sent an unsigned request to an operation declared in `get_adcp_capabilities.request_signing.required_for`. Your agent rejected with 401 but did not include a `WWW-Authenticate: Signature ...` challenge header, so the runner could not resolve the error code from the transport binding. + +**Fix:** Emit the RFC 9421 challenge header on every 401 caused by missing or invalid signatures. The runner resolves the error code via the transport binding order — if the `WWW-Authenticate` header is absent, the error classification falls back to "(none)" even when the JSON body carries a useful message. + +The reference SDK constructs these errors via `RequestSignatureError` from `@adcp/client/signing` with `.code: RequestSignatureErrorCode`. The full taxonomy (`request_signature_required`, `request_signature_header_malformed`, `request_signature_tag_invalid`, `request_signature_window_invalid`, `request_signature_key_unknown`, etc.) is enumerated in that module. Your agent SHOULD surface the same code on the challenge so SDK-speaking callers can recover automatically. + +See [Signed Requests (Transport Layer)](/docs/building/implementation/security#signed-requests-transport-layer) for the challenge-header format and the [transport-error binding order](/docs/building/implementation/transport-errors). + +## Response envelope drift + +``` +× (unknown step): Response contains errors array +``` + +The vector used `check: error_code` but your response surfaces the error on a shape the runner's client-detection order didn't expect. In practice, this means your agent returned `errors[]` when the transport layer already carried `adcp_error`, or vice versa — the storyboard asserted a single error code and the runner resolved it from a different layer than you emitted on. + +**Fix:** Pick one error surface per response and stick to it per the [envelope vs. payload two-layer model](/docs/building/implementation/error-handling#envelope-vs-payload-errors-the-two-layer-model). On MCP: `adcp_error` for structured content; `errors[]` for task-payload errors. On A2A: the same layers apply — transport error in the envelope, application error in the task artifact's DataPart. + +The runner's `check: error_code` is shape-agnostic — it resolves from either layer — but if your agent emits both simultaneously they can disagree, and the runner grades the resolved code against the vector's expectation. Choosing one surface and being consistent avoids the divergence. + +## Context echo failures + +``` +× (unknown step): expected field "context.correlation_id" = "xyz", got (missing) +``` + +Your agent returned a response that does not include the `context:` object from the request. Every storyboard step that sends `context: { correlation_id: ... }` asserts that `context.correlation_id` echoes unchanged in the response. + +**Fix:** Preserve the full `context:` object verbatim on every response, including errors. The echo contract is normative — buyers use `correlation_id` to stitch multi-agent flows, and the runner grades every context-carrying step on it. See [Context and sessions — Normative echo contract](/docs/building/integration/context-sessions#normative-echo-contract). + +Captures use the same contract in reverse: storyboards that pass `"$context."` through `context_outputs:` rely on the capture populating after the producer step's validations pass. A downstream step reading `$context.foo` when the producer failed or omitted `context:` grades as `unresolved_substitution`. + +## Capability-vector mismatch (runner declared, agent doesn't support) + +``` +× (unknown step): capability X asserted but not declared in get_adcp_capabilities +``` + +The storyboard dispatched a step that requires a capability your agent does not advertise in its `get_adcp_capabilities` response. The runner should auto-skip these steps; if you're seeing them graded as failures instead, either the capability is declared at the wrong key or the runner is missing the auto-skip path. + +**Fix:** Double-check your `get_adcp_capabilities.tools` list and any required-for fields (`request_signing.required_for`, `idempotency.supported_tools`, etc.). For vectors that apply only to specialized agents, the storyboard author can use `skipVectors` to flag the opt-out explicitly; as an implementer, the fix is almost always on the capability declaration rather than the vector. + +## Required-for composition + +``` +× (unknown step): missing auth — step requires authenticated or signed +``` + +The runner encountered a mutating step that expected either authenticated credentials or a signed request, and the transport carried neither. Typically this means the test kit didn't declare `auth.api_key` AND the agent doesn't advertise request-signing support — leaving the runner with no way to authenticate the call. + +**Fix:** Either (a) declare `auth.api_key` in the test kit so the runner uses Bearer auth, or (b) advertise request-signing via `get_adcp_capabilities.request_signing` so the runner signs the request instead. The runner's `requireAuthenticatedOrSigned` gate accepts either path — it fails only when both are absent. + +## `INVALID_STATE` vs `INVALID_TRANSITION` + +Two codes that are easy to confuse: + +- **`INVALID_STATE`** — the canonical AdCP media-buy error code for "the resource is in a state that doesn't allow this action." Used on `create_media_buy`/`update_media_buy`/`pause`/`resume`/`cancel` against a media buy that cannot transition as requested. See `media-buy/specification.mdx` and `media-buy/media-buys/index.mdx` for authoritative usage. +- **`INVALID_TRANSITION`** — specific to the `comply_test_controller` sandbox primitive. Emitted when a runner requests a state-machine transition the seller rejects (e.g., forcing `approved` → `archived` without going through `active`). See [Compliance test controller — Scenarios](/docs/building/implementation/comply-test-controller#scenarios). + +A storyboard vector that asserts `INVALID_STATE` on a production task but your agent returns `INVALID_TRANSITION` is an error-code vocabulary mismatch — `INVALID_TRANSITION` is not in the canonical enum at `static/schemas/source/enums/error-code.json` and should not appear outside the compliance test controller. + +## When none of the above matches + +If you hit a failure that doesn't map to anything here, check the [known spec ambiguities](/docs/building/implementation/known-ambiguities) page — some storyboards are blocked on resolved-but-unreleased spec gaps, and the workaround is tracked there. + +Still stuck? File an issue at [adcontextprotocol/adcp](https://github.com/adcontextprotocol/adcp/issues) with the full runner output and the storyboard name. Maintainers can usually narrow the pattern from the error signature. diff --git a/docs/building/validate-your-agent.mdx b/docs/building/validate-your-agent.mdx index 60266f5894..3a262524af 100644 --- a/docs/building/validate-your-agent.mdx +++ b/docs/building/validate-your-agent.mdx @@ -213,6 +213,11 @@ For [Practitioner certification](https://agenticadvertising.org/certification), All commands support `--json`, `--debug`, `--auth TOKEN`, and `--protocol mcp|a2a`. +## When a storyboard fails + +- **[Storyboard troubleshooting](/docs/building/implementation/storyboard-troubleshooting)** — Error patterns mapped to root causes and fixes (missing fixtures, signature challenges, envelope drift, context echo, capability mismatches) +- **[Known spec ambiguities](/docs/building/implementation/known-ambiguities)** — Open spec gaps that affect conformance, with workarounds and issue links + ## What's next - **[Compliance test controller](/docs/building/implementation/comply-test-controller)** — Implement deterministic testing for full lifecycle coverage diff --git a/static/compliance/source/universal/idempotency.yaml b/static/compliance/source/universal/idempotency.yaml index f40c7ccd5c..77e7a19aec 100644 --- a/static/compliance/source/universal/idempotency.yaml +++ b/static/compliance/source/universal/idempotency.yaml @@ -112,6 +112,16 @@ phases: A create_media_buy request without idempotency_key MUST be rejected. The schema marks the field as required, so this is typically caught at validation time. + The reference `@adcp/client` SDK auto-injects `idempotency_key` on mutating + tasks via `applyIdempotencyInvariant` + client-side shaping. Without an + explicit opt-out this vector would never reach the agent with a missing key + — the runner would inject one before dispatch, and the vector would silently + pass for every SDK-speaking agent regardless of enforcement. The step below + sets `omit_idempotency_key: true` so the runner skips both its own + invariant application and the SDK's auto-inject. Do not remove the flag: + without it, "certified" agents all carry an unverified assertion on this + vector. + steps: - id: create_media_buy_missing_key title: "Missing idempotency_key returns INVALID_REQUEST" @@ -125,6 +135,7 @@ phases: doc_ref: "/media-buy/task-reference/create_media_buy" comply_scenario: error_handling expect_error: true + omit_idempotency_key: true stateful: false expected: | Reject with: diff --git a/static/compliance/source/universal/storyboard-schema.yaml b/static/compliance/source/universal/storyboard-schema.yaml index 983342d84b..67ae22e8cf 100644 --- a/static/compliance/source/universal/storyboard-schema.yaml +++ b/static/compliance/source/universal/storyboard-schema.yaml @@ -136,6 +136,13 @@ # → use a per-run random JWT-shaped bogus Bearer token. # Runner generates `..` per run. # +# omit_idempotency_key: boolean (optional — when true on a mutating-task step, +# the runner skips `applyIdempotencyInvariant` and signals the SDK client via +# `skipIdempotencyAutoInject` so the request reaches the agent WITHOUT an +# `idempotency_key`. Use for missing-key rejection vectors — without this flag +# the SDK auto-injects a key and the agent never sees the missing-key path. +# Applies only to mutating tasks (create/update/cancel); ignored on reads.) +# # contributes_to: string (optional — marks this step as contributing a named flag # that a later assert_contribution step can require) # contributes_if: string (optional — runner-evaluated expression gating the contribution)