Skip to content

feat(testing): createComplyController seller scaffold (#701)#708

Merged
bokelley merged 2 commits into
mainfrom
bokelley/issue-701
Apr 21, 2026
Merged

feat(testing): createComplyController seller scaffold (#701)#708
bokelley merged 2 commits into
mainfrom
bokelley/issue-701

Conversation

@bokelley

Copy link
Copy Markdown
Contributor

Summary

  • Adds createComplyController to @adcp/client/testing — a domain-grouped (seed / force / simulate) seller-side scaffold for AdCP's comply_test_controller tool. Returns { toolDefinition, handle, handleRaw, register } so sellers wire the tool with a single controller.register(server) call.
  • Extends TestControllerStore with five seed methods (seedProduct, seedPricingOption, seedCreative, seedPlan, seedMediaBuy). Both the existing registerTestController flat-store surface and the new grouped façade now cover the full 12-scenario protocol.
  • Hardened against common misuse: strict === true gate, capped seed cache, prototype-key rejection, frozen tool definition, list_scenarios bypasses the gate so capability probes always succeed, and register() warns once per controller when no gate and no env opt-out is configured.

Closes #701.

What the helper owns (so the seller doesn't)

  1. Scenario dispatch — routes scenario to the correct adapter; unregistered adapters return UNKNOWN_SCENARIO.
  2. Param validation — per-scenario required fields + status-enum parsing; rejections return INVALID_PARAMS.
  3. Seed idempotency — same id + equivalent fixture = previous_state: "existing"; divergent fixture = INVALID_PARAMS without touching the adapter.
  4. Error envelope shaping — adapters throw TestControllerError(code, msg, currentState?) or return the success shape; the helper never leaks unexpected errors.
  5. Sandbox gating — optional sandboxGate with strict === true semantics, fails closed on throws, bypassed for list_scenarios.
  6. MCP envelopecontent + structuredContent + isError via the existing toMcpResponse.

What stays adapter-side (deliberate)

  • Transition enforcement. The spec requires controllers and production to share one state machine; the helper calls into the adapter's rules rather than shipping a built-in policy that could silently diverge. The example shows the pattern with a CREATIVE_TRANSITIONS table the adapter consults.

Scope trimmed from #701 (follow-up issues filed)

Test plan

  • 36 new unit tests in test/comply-controller.test.js covering dispatch, list_scenarios, seed idempotency (fresh / equivalent / divergent / cross-kind / cache-eviction race), sandbox gate (allow / deny / throw / truthy-non-boolean / list_scenarios bypass), fixture hardening (__proto__ / constructor / prototype / non-object types), seed cache cap, register warning de-dupe + env opt-outs, and MCP envelope shape.
  • 61 existing test/server-test-controller.test.js pass unchanged (back-compat).
  • Full repo suite: 0 failures (6 pre-existing skips).
  • TypeScript compile clean (tsc --noEmit); lint clean on new files.
  • Built-output sanity check: require('./dist/lib/testing') resolves createComplyController, SEED_SCENARIOS, createSeedFixtureCache.
  • CI green.

🤖 Generated with Claude Code

bokelley and others added 2 commits April 21, 2026 09:21
…_controller (#701)

Add `createComplyController` to `@adcp/client/testing` — a domain-grouped
(seed / force / simulate) seller-side scaffold for the comply_test_controller
tool. Also extends TestControllerStore with seed methods and exports
SeedFixtureCache / createSeedFixtureCache / SEED_SCENARIOS from the existing
flat-store surface so both APIs support the full 12-scenario protocol.

Hardened against common misuse: sandbox gate requires strict === true; seed
cache is capped at 1000 net-new keys; fixture keys __proto__ / constructor /
prototype are rejected; toolDefinition is frozen with a shallow-copied
inputSchema; list_scenarios bypasses the gate so capability probes always
succeed; register() warns when no gate is configured and no opt-out env flag
is set. Closes #701.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@bokelley bokelley merged commit 46de887 into main Apr 21, 2026
12 checks passed
bokelley added a commit that referenced this pull request Apr 21, 2026
…ontroller skill refs

Matrix perf:
- New `--shared-node-modules <path>` flag on `agent-skill-storyboard.ts`
  symlinks the scratch workspace's `node_modules` to a cached template
  instead of running `npm install` per pair. The matrix driver prepares
  the template once in `.cache/harness-template/` (~30s) and every
  subsequent pair skips install (~20s saved per pair).
- `run-skill-matrix.ts` parallel default auto-picks `min(4, cpus/2)` —
  was `1`. 14-pair full matrix drops from ~34 min serial to ~9 min on
  a typical laptop.
- Fast-fail up-front if the `claude` CLI isn't on PATH (saves every
  pair from hitting the same wall).
- Summary line now prints wall, cpu, and speedup so regressions are
  obvious.
- `--no-shared-install` escape hatch for debugging template drift.

Skills (createComplyController is the new scaffold from #708):
- build-seller-agent/SKILL.md § Compliance Testing now leads with
  `createComplyController` (adapter-based, handles dispatch +
  idempotency + sandbox gating). `registerTestController` demoted to
  the "low-level alternative" footnote.
- build-creative-agent/SKILL.md gains § Deterministic Testing covering
  seed.creative + force.creative_status adapters, required for
  `creative_generative/seller` and `deterministic_testing` storyboards.
- docs/guides/VALIDATE-YOUR-AGENT.md gains § Deterministic testing
  section pointing at `createComplyController` with the canonical
  adapter example.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bokelley added a commit that referenced this pull request Apr 21, 2026
…l 8 skills (#709)

* docs(validation): VALIDATE-YOUR-AGENT index + skill-matrix harness

New operator-facing docs/guides/VALIDATE-YOUR-AGENT.md: five-command
checklist plus command-by-command references for storyboard runner,
adcp fuzz (T1/T2/T3), adcp grade request-signing, multi-instance,
webhook conformance, schema-driven validation hooks, custom
--invariants, and SubstitutionEncoder/Observer. BUILD-AN-AGENT.md and
root CLAUDE.md link to it so builders find the full surface without
reading source.

New scripts/manual-testing/skill-matrix.json + run-skill-matrix.ts
driver, exposed as `npm run compliance:skill-matrix`. Fans out the
existing agent-skill-storyboard harness across 14 skill × storyboard
pairs (seller, brand-rights, creative, generative-seller, governance,
retail-media, si, signals) with --filter, --parallel, --stop-on-first-fail.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(skills): unified § Validate Locally section across all 8 build-*-agent

Replaces skill-specific § Validation sections (which varied in depth and
didn't mention fuzz, webhook-receiver, multi-instance, substitution,
--invariants, or the skill-matrix harness) with a consistent § Validate
Locally block per skill. Each block points at docs/guides/VALIDATE-YOUR-AGENT.md
for the full picture, then covers specialism-specific commands: canonical
storyboard IDs, cross-cutting bundles (security_baseline, idempotency,
schema_validation, error_compliance), rejection-surface fuzz with the
right --tools list, and a decoder for the most common per-specialism
failure modes.

Skill-specific extras:
- retail-media: SubstitutionEncoder.encode_for_url_context wiring for
  catalog-driven macro URLs; mentions substitution_encoding_violation /
  substitution_binding_missing grader failures.
- seller: specialism bundle enumeration (sales_guaranteed, broadcast_tv,
  streaming_tv, social, etc.); signed-requests grader pointer; multi-
  instance pointer.
- governance: --auto-seed fuzz (Tier 3 update-tool coverage); 3.0 GA
  authority_level → human_review_required reminder.

Every skill also points at `npm run compliance:skill-matrix -- --filter <name>`
as the port-less fallback — invokes the dogfood harness in an isolated
workspace when local binding isn't available.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: prettier + changeset for PR #709

* feat(harness): shared node_modules + parallel default + createComplyController skill refs

Matrix perf:
- New `--shared-node-modules <path>` flag on `agent-skill-storyboard.ts`
  symlinks the scratch workspace's `node_modules` to a cached template
  instead of running `npm install` per pair. The matrix driver prepares
  the template once in `.cache/harness-template/` (~30s) and every
  subsequent pair skips install (~20s saved per pair).
- `run-skill-matrix.ts` parallel default auto-picks `min(4, cpus/2)` —
  was `1`. 14-pair full matrix drops from ~34 min serial to ~9 min on
  a typical laptop.
- Fast-fail up-front if the `claude` CLI isn't on PATH (saves every
  pair from hitting the same wall).
- Summary line now prints wall, cpu, and speedup so regressions are
  obvious.
- `--no-shared-install` escape hatch for debugging template drift.

Skills (createComplyController is the new scaffold from #708):
- build-seller-agent/SKILL.md § Compliance Testing now leads with
  `createComplyController` (adapter-based, handles dispatch +
  idempotency + sandbox gating). `registerTestController` demoted to
  the "low-level alternative" footnote.
- build-creative-agent/SKILL.md gains § Deterministic Testing covering
  seed.creative + force.creative_status adapters, required for
  `creative_generative/seller` and `deterministic_testing` storyboards.
- docs/guides/VALIDATE-YOUR-AGENT.md gains § Deterministic testing
  section pointing at `createComplyController` with the canonical
  adapter example.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: regen agent docs (creative_generative flow)

* chore: regen docs/llms.txt after upstream sync (57 storyboards)

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(testing): SDK scaffold for comply_test_controller handler

1 participant