From 0edc857ce1953fdc04a22e3f66bca474a1b398c1 Mon Sep 17 00:00:00 2001 From: Brian O'Kelley Date: Sat, 16 May 2026 14:55:21 -0400 Subject: [PATCH 1/2] docs(compliance): name the verifiability gap and the SDK bridge's role (#4593) Adds a "Test surfaces and the verifiability gap" section to the comply_test_controller doc, framing the gap as universal across seller classes. State-local sellers close it via DB-backed `seed_*`; upstream-proxy sellers close it via the SDK's `TestControllerBridge`. The bridge is one implementation of the test-surface pattern, not a separate seller category. Documents the non-normative `_bridge` marker (shipped in adcp-client#1786) as the response-level signal runners read to distinguish fixture-merged content from upstream-derived content, and adds a three-axis table covering test mode, the `account.sandbox` flag, and bridge participation. Closes #4593. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../4593-test-surfaces-verifiability-gap.md | 6 +++ .../by-layer/L3/comply-test-controller.mdx | 38 +++++++++++++++++++ 2 files changed, 44 insertions(+) create mode 100644 .changeset/4593-test-surfaces-verifiability-gap.md diff --git a/.changeset/4593-test-surfaces-verifiability-gap.md b/.changeset/4593-test-surfaces-verifiability-gap.md new file mode 100644 index 0000000000..31cec736db --- /dev/null +++ b/.changeset/4593-test-surfaces-verifiability-gap.md @@ -0,0 +1,6 @@ +--- +--- + +docs(compliance): name the verifiability gap and the role of the SDK bridge in the comply_test_controller doc (#4593). + +Adds a "Test surfaces and the verifiability gap" section before the existing "Compliance testing modes" coverage. Frames the gap as universal — every seller faces it — and describes the two implementations (DB-backed `seed_*` for state-local sellers, SDK `TestControllerBridge` for upstream-proxy sellers) as different paths to the same goal, not different seller categories. Documents the non-normative `_bridge` marker (shipped in adcp-client#1786) as the response-level signal that lets runners distinguish fixture-merged content from upstream-derived content. Adds a three-axis disambiguation table covering test mode, the `account.sandbox` flag, and bridge participation. diff --git a/docs/building/by-layer/L3/comply-test-controller.mdx b/docs/building/by-layer/L3/comply-test-controller.mdx index 878fc2f9b9..adff61e12e 100644 --- a/docs/building/by-layer/L3/comply-test-controller.mdx +++ b/docs/building/by-layer/L3/comply-test-controller.mdx @@ -611,6 +611,44 @@ State transition scenarios (`force_*`) are idempotent: forcing a status that mat Simulation scenarios (`simulate_*`) are NOT idempotent — `simulate_delivery` adds to existing totals, while `simulate_budget_spend` replaces the current spend level. +## Test surfaces and the verifiability gap + +Every seller faces the same compliance question: can a runner verify the seller's wire format separately from the seller's live behavior? + +- **Wire format** can be tested against fixtures. Storyboards exercise request and response shapes; pass/fail is mechanical. +- **Live behavior** — does the seller's adapter actually reach the system that owns its state? — can only be tested against a real test surface (sandbox credentials, a wired adapter to upstream, real catalog data, real account state). + +These are independent reliability claims, and the AAO Verified mark surfaces both axes via [`(Spec)` and `(Sandbox)` qualifiers](/docs/building/verification/aao-verified). The verifiability gap — the second axis — is universal. Where a seller's state-of-record lives determines *how* they close it, not whether they have to: + +| Where state-of-record lives | How the test loop closes | +|---|---| +| Local DB only | `comply_test_controller.seed_*` writes to the seller's DB; the seller's read handlers consume the same store. The seed→read loop closes naturally. | +| Upstream system the seller does not control | Seeded writes are dead to the read handler. The TypeScript SDK ships a `TestControllerBridge` that runs the real adapter call first (so a broken upstream call still fails the gate), then merges seeded fixtures into the response. | +| Mixed (local DB for some tools, upstream for others) | Both, per tool. | + +The bridge is **one implementation** of the test-surface pattern, not a separate seller category. A state-local seller without wired seeds and an upstream-proxy seller without a wired bridge are in the same epistemic position: verifiable wire format, unverifiable live behavior. Naming the underlying gap matters; classifying sellers by where their state lives does not. + +### The `_bridge` marker + +When a response passes through the SDK's `TestControllerBridge`, the SDK stamps a non-normative `_bridge: { callback, tool, merged_count }` marker on the response so runners can distinguish wire conformance against fixture data from adapter health against upstream: + +- **Marker present on a step.** The response content was merged from a seeded fixture after the seller's handler returned. The storyboard still proves wire-format conformance for that step; it does not prove the seller's adapter reached its upstream system. +- **Marker absent on every step of a storyboard.** Every response came from the seller's adapter end-to-end (or from a local DB the runner seeded directly). The storyboard proves both wire conformance and live behavior for the tools it exercises. + +The marker is **non-normative** — sellers MUST NOT depend on its presence or absence in production responses, and runners treat the marker as advisory metadata, not as a conformance signal in its own right. Its purpose is to let downstream leaderboards and the AAO compliance heartbeat distinguish "this pass exercised the adopter's adapter against upstream" from "this pass exercised wire conformance against fixture data the SDK merged." Marker design is tracked in [`adcp-client#1775`](https://github.com/adcontextprotocol/adcp-client/issues/1775); the marker itself shipped in [`adcp-client#1786`](https://github.com/adcontextprotocol/adcp-client/pull/1786); the leaderboard policy that consumes it lives in [`adcp-client#1782`](https://github.com/adcontextprotocol/adcp-client/issues/1782). + +### Three orthogonal axes + +The doc has three test-related controls that adopters often conflate. They answer different questions: + +| Axis | Question it answers | +|---|---| +| **Test mode** (`comply_test_controller`) | "Is a compliance runner driving this test?" | +| **Sandbox flag** (`account.sandbox`) | "Is the targeted account a sandbox account, with no real-world side effects?" | +| **Bridge participation** (`_bridge` marker on a response) | "Did this response come from the adapter's upstream call, or from a fixture the SDK merged in?" | + +A storyboard pass can have any combination of these signals. The first determines whether deterministic forces are allowed; the second guards real-world side effects on the seller's prod stack; the third disambiguates the source of response content within a passing storyboard. + ## Compliance testing modes The presence of `comply_test_controller` in a seller's tool list determines which mode a compliance tester uses: From 3c085c04eb5234cb2f0254fa3766e9d10e1b8041 Mon Sep 17 00:00:00 2001 From: Brian O'Kelley Date: Sat, 16 May 2026 15:17:53 -0400 Subject: [PATCH 2/2] docs(conformance): move test-surfaces framing to conformance.mdx per expert feedback MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Restructures PR #4619 in response to convergent DX/docs/product/protocol-expert review: - Canonical home for the framing is `conformance.mdx`, not the L3 controller page. The audience for this framing (DSP/SSP engineers, RMN proxy implementers, procurement leads reading badge docs) is broader than the audience for the controller tool definition. - Drops the "live behavior" wording to avoid re-importing the deprecated (Live) framing rejected in #4379 — (Sandbox) attests sandbox-flag tolerance on prod, not adapter-to-upstream calls. - "Three orthogonal axes" → "Three signals — don't conflate them" so the table doesn't collide with the (Spec)/(Sandbox) "two axes" language already load-bearing in `conformance.mdx` and `aao-verified.mdx`. - "Test mode" axis label → "Test controller availability" to disambiguate from BidRequest-style test flags an OpenRTB reviewer would expect. - Defines "state-local sellers" and "upstream-proxy sellers" inline on first use (SSPs/creative-agents; DSPs/retail-media/signals-brokers). - Pins the leading-underscore convention for SDK/runner-stamped metadata reserved for testing tooling, so future SDK fields can't pollute the wire surface without going through the same gate. - Leaves a short pointer on the L3 controller page back to the canonical section, and adds a reciprocal hook in `aao-verified.mdx`'s existing controller-relationship Note. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../4593-test-surfaces-verifiability-gap.md | 4 +- .../by-layer/L3/comply-test-controller.mdx | 38 ++----------------- docs/building/verification/aao-verified.mdx | 2 +- docs/building/verification/conformance.mdx | 30 +++++++++++++++ 4 files changed, 36 insertions(+), 38 deletions(-) diff --git a/.changeset/4593-test-surfaces-verifiability-gap.md b/.changeset/4593-test-surfaces-verifiability-gap.md index 31cec736db..edb1783984 100644 --- a/.changeset/4593-test-surfaces-verifiability-gap.md +++ b/.changeset/4593-test-surfaces-verifiability-gap.md @@ -1,6 +1,6 @@ --- --- -docs(compliance): name the verifiability gap and the role of the SDK bridge in the comply_test_controller doc (#4593). +docs(conformance): document test surfaces, the SDK bridge, and the `_bridge` marker (#4593). -Adds a "Test surfaces and the verifiability gap" section before the existing "Compliance testing modes" coverage. Frames the gap as universal — every seller faces it — and describes the two implementations (DB-backed `seed_*` for state-local sellers, SDK `TestControllerBridge` for upstream-proxy sellers) as different paths to the same goal, not different seller categories. Documents the non-normative `_bridge` marker (shipped in adcp-client#1786) as the response-level signal that lets runners distinguish fixture-merged content from upstream-derived content. Adds a three-axis disambiguation table covering test mode, the `account.sandbox` flag, and bridge participation. +Adds a "Test surfaces and the storyboard loop" section to the Conformance Specification covering the two implementations of the test-surface pattern — DB-backed `seed_*` for state-local sellers (SSPs, creative agents) and the TypeScript SDK's `TestControllerBridge` for upstream-proxy sellers (DSPs, retail-media networks, signals brokers). Frames both as the same pattern, not different seller categories, and clarifies that both earn `(Spec)` while neither is what `(Sandbox)` attests. Documents the SDK's non-normative `_bridge` response marker (shipped in adcp-client#1786) and pins the underscore-prefix convention for SDK/runner-stamped metadata reserved for testing tooling. Adds a three-signal disambiguation table covering test controller availability, the `account.sandbox` flag, and `_bridge` participation. The `comply_test_controller` doc keeps a short pointer back to the canonical section, and the AAO Verified doc cross-links into it from the existing controller-relationship Note. diff --git a/docs/building/by-layer/L3/comply-test-controller.mdx b/docs/building/by-layer/L3/comply-test-controller.mdx index adff61e12e..ca5f35bcb6 100644 --- a/docs/building/by-layer/L3/comply-test-controller.mdx +++ b/docs/building/by-layer/L3/comply-test-controller.mdx @@ -611,43 +611,11 @@ State transition scenarios (`force_*`) are idempotent: forcing a status that mat Simulation scenarios (`simulate_*`) are NOT idempotent — `simulate_delivery` adds to existing totals, while `simulate_budget_spend` replaces the current spend level. -## Test surfaces and the verifiability gap +## Test surfaces -Every seller faces the same compliance question: can a runner verify the seller's wire format separately from the seller's live behavior? +Where a seller's state-of-record lives determines how the storyboard test loop closes. State-local sellers (typically SSPs, creative agents) write to the seller's DB via the `seed_*` scenarios above; the seller's read handlers consume the same store, and the seed→read loop closes naturally. Upstream-proxy sellers (DSPs proxying to platforms, retail-media networks reading retailer catalogs, signals brokers) cannot close the loop that way because their read handlers reach a system the seller does not control; the TypeScript SDK ships a `TestControllerBridge` that runs the real adapter call first, then merges seeded fixtures into the response. Either path earns the wire-format pass that `AAO Verified (Spec)` attests. Neither path is what `(Sandbox)` attests — that's a separate axis covering whether the seller's production stack honors `account.sandbox: true` without real-world side effects. -- **Wire format** can be tested against fixtures. Storyboards exercise request and response shapes; pass/fail is mechanical. -- **Live behavior** — does the seller's adapter actually reach the system that owns its state? — can only be tested against a real test surface (sandbox credentials, a wired adapter to upstream, real catalog data, real account state). - -These are independent reliability claims, and the AAO Verified mark surfaces both axes via [`(Spec)` and `(Sandbox)` qualifiers](/docs/building/verification/aao-verified). The verifiability gap — the second axis — is universal. Where a seller's state-of-record lives determines *how* they close it, not whether they have to: - -| Where state-of-record lives | How the test loop closes | -|---|---| -| Local DB only | `comply_test_controller.seed_*` writes to the seller's DB; the seller's read handlers consume the same store. The seed→read loop closes naturally. | -| Upstream system the seller does not control | Seeded writes are dead to the read handler. The TypeScript SDK ships a `TestControllerBridge` that runs the real adapter call first (so a broken upstream call still fails the gate), then merges seeded fixtures into the response. | -| Mixed (local DB for some tools, upstream for others) | Both, per tool. | - -The bridge is **one implementation** of the test-surface pattern, not a separate seller category. A state-local seller without wired seeds and an upstream-proxy seller without a wired bridge are in the same epistemic position: verifiable wire format, unverifiable live behavior. Naming the underlying gap matters; classifying sellers by where their state lives does not. - -### The `_bridge` marker - -When a response passes through the SDK's `TestControllerBridge`, the SDK stamps a non-normative `_bridge: { callback, tool, merged_count }` marker on the response so runners can distinguish wire conformance against fixture data from adapter health against upstream: - -- **Marker present on a step.** The response content was merged from a seeded fixture after the seller's handler returned. The storyboard still proves wire-format conformance for that step; it does not prove the seller's adapter reached its upstream system. -- **Marker absent on every step of a storyboard.** Every response came from the seller's adapter end-to-end (or from a local DB the runner seeded directly). The storyboard proves both wire conformance and live behavior for the tools it exercises. - -The marker is **non-normative** — sellers MUST NOT depend on its presence or absence in production responses, and runners treat the marker as advisory metadata, not as a conformance signal in its own right. Its purpose is to let downstream leaderboards and the AAO compliance heartbeat distinguish "this pass exercised the adopter's adapter against upstream" from "this pass exercised wire conformance against fixture data the SDK merged." Marker design is tracked in [`adcp-client#1775`](https://github.com/adcontextprotocol/adcp-client/issues/1775); the marker itself shipped in [`adcp-client#1786`](https://github.com/adcontextprotocol/adcp-client/pull/1786); the leaderboard policy that consumes it lives in [`adcp-client#1782`](https://github.com/adcontextprotocol/adcp-client/issues/1782). - -### Three orthogonal axes - -The doc has three test-related controls that adopters often conflate. They answer different questions: - -| Axis | Question it answers | -|---|---| -| **Test mode** (`comply_test_controller`) | "Is a compliance runner driving this test?" | -| **Sandbox flag** (`account.sandbox`) | "Is the targeted account a sandbox account, with no real-world side effects?" | -| **Bridge participation** (`_bridge` marker on a response) | "Did this response come from the adapter's upstream call, or from a fixture the SDK merged in?" | - -A storyboard pass can have any combination of these signals. The first determines whether deterministic forces are allowed; the second guards real-world side effects on the seller's prod stack; the third disambiguates the source of response content within a passing storyboard. +The cross-page framing for both implementations of this pattern, the SDK's `_bridge` advisory marker, and the runtime-signals disambiguation table all live in the Conformance Specification → [Test surfaces and the storyboard loop](/docs/building/verification/conformance#test-surfaces-and-the-storyboard-loop). ## Compliance testing modes diff --git a/docs/building/verification/aao-verified.mdx b/docs/building/verification/aao-verified.mdx index cd516c00a6..63370e2288 100644 --- a/docs/building/verification/aao-verified.mdx +++ b/docs/building/verification/aao-verified.mdx @@ -57,7 +57,7 @@ The seller-side gate is normative: every comply_test_controller request includes The (Sandbox) qualifier replaces the earlier draft's `Verified (Live)` framing. The change: instead of attesting "your real-money production code path delivers impressions correctly" (canonical campaigns running through your stack), (Sandbox) attests "your real production code path correctly handles sandbox-flagged traffic across the full storyboard suite." Both are real-prod-surface claims; the difference is what gets tested. (Sandbox) is universally achievable across specialisms with no new AAO operational infrastructure. See [#4379](https://github.com/adcontextprotocol/adcp/issues/4379) for the reframe verdict. -**Re: `comply_test_controller`**: the controller is a **dev/staging-only** affordance for adopters' own integration testing. AAO's (Sandbox) grading does not require or use it. Sellers MAY implement controller endpoints in their dev environment to support deterministic local testing, but the production stack does not need to expose `comply_test_controller` to earn (Sandbox). The seller-side sandbox gate is what (Sandbox) attests — schema and lifecycle correctness under flagged traffic, on real prod. +**Re: `comply_test_controller`**: the controller is a **dev/staging-only** affordance for adopters' own integration testing. AAO's (Sandbox) grading does not require or use it. Sellers MAY implement controller endpoints in their dev environment to support deterministic local testing, but the production stack does not need to expose `comply_test_controller` to earn (Sandbox). The seller-side sandbox gate is what (Sandbox) attests — schema and lifecycle correctness under flagged traffic, on real prod. How the dev-time test surface itself is stood up — DB-backed `seed_*` for state-local sellers vs the SDK's `TestControllerBridge` for upstream-proxy sellers — is covered in [Test surfaces and the storyboard loop](/docs/building/verification/conformance#test-surfaces-and-the-storyboard-loop). ## Naming history diff --git a/docs/building/verification/conformance.mdx b/docs/building/verification/conformance.mdx index 3b0cb0284b..ae35463368 100644 --- a/docs/building/verification/conformance.mdx +++ b/docs/building/verification/conformance.mdx @@ -30,6 +30,36 @@ A second axis — **AAO Verified (Sandbox)** — verifies the seller's real prod The two qualifiers share one brand mark — **AAO Verified** — and an agent can earn either or both. **(Spec) and (Sandbox) are independent**: each independently demonstrates conformance through different evidence. (Spec) attests wire-format conformance against any registered endpoint; (Sandbox) attests the production code path correctly tolerates sandbox-flagged traffic. See [AAO Verified](/docs/building/verification/aao-verified) for the qualifier model and the [Sandbox framing verdict](https://github.com/adcontextprotocol/adcp/issues/4379); the rest of this page indexes the storyboards that back both qualifiers. +## Test surfaces and the storyboard loop + +Every seller exposes a *test surface* — the mechanism that lets a storyboard runner exercise the seller's tools deterministically without triggering real-world side effects. The test surface is what (Spec) is graded against. How a seller stands up that surface depends on where their state-of-record lives; the implementation differs, the goal does not: + +| Where state-of-record lives | How the test loop closes | +|---|---| +| Local DB only (typically SSPs, creative agents) | The storyboard runner writes fixtures via `comply_test_controller.seed_*`; the seller's read handlers consume the same store. The seed → read loop closes naturally. | +| Upstream system the seller does not control (DSPs proxying to platforms, retail-media networks reading retailer catalogs, signals brokers) | Seeded writes are dead to the read handler. The TypeScript SDK ships a `TestControllerBridge` that runs the real adapter call first (so a broken upstream call still fails the gate), then merges seeded fixtures into the response. | +| Mixed (some tools local, some upstream) | Both, per tool. | + +Both paths earn `(Spec)` — both prove the seller's wire format matches the storyboards. The bridge is **one implementation** of the test-surface pattern, not a separate seller category. A state-local seller without wired seeds and an upstream-proxy seller without a wired bridge are in the same position: storyboards cannot run end-to-end against them. Neither category is what `(Sandbox)` attests; `(Sandbox)` is the separate axis covering whether the seller's production stack honors `account.sandbox: true` without real-world side effects. + +### Distinguishing fixture-merged from upstream-derived responses + +When a response passes through the SDK's `TestControllerBridge`, the SDK stamps a `_bridge: { callback, tool, merged_count }` marker on the response. Marker presence on a step means the response content was merged from a seeded fixture after the seller's handler returned; marker absence means the response came from the seller's adapter end-to-end (or from a local DB the runner seeded directly). The marker is advisory metadata for runners and downstream leaderboards — it is **not** part of the wire contract. Sellers MUST NOT emit it, and conformance checks ignore it. The leading underscore marks the field as SDK/runner-stamped metadata reserved for testing tooling; future fields with the same prefix follow the same rule. + +Marker design: [`adcp-client#1775`](https://github.com/adcontextprotocol/adcp-client/issues/1775). Shipped: [`adcp-client#1786`](https://github.com/adcontextprotocol/adcp-client/pull/1786). Leaderboard policy that consumes the marker: [`adcp-client#1782`](https://github.com/adcontextprotocol/adcp-client/issues/1782). + +### Three signals — don't conflate them + +Adopters often read these three controls as the same thing. They answer different questions: + +| Signal | Question it answers | +|---|---| +| Test controller availability (`comply_test_controller` in `tools/list`) | "Has the seller exposed deterministic-mode forces?" | +| Sandbox flag (`account.sandbox` on requests) | "Is the targeted account a sandbox account, with no real-world side effects?" | +| Bridge participation (`_bridge` marker on a response) | "Did this response come from the adapter's upstream call, or from a fixture the SDK merged in?" | + +These are **runtime controls** on individual storyboard steps — distinct from the `(Spec)` and `(Sandbox)` verification qualifiers, which describe what a storyboard pass *attests* over time. A storyboard pass can carry any combination of the three signals. + ## Storyboards are the truth Rather than restate every MUST in prose — which would inevitably drift from the executable suite — **the storyboards ARE the conformance specification.** This document is a navigational index to them, grouped by the declaration that obligates the storyboard to run.