feat(training-agent): storyboard pass — signing composition, review polish, RFC 9728, dual-mode CI#2582
Merged
Merged
Conversation
- Export getWebhookSigningKey() from webhooks.ts so the future
createAdcpServer({ webhooks: { signerKey } }) config can reuse the
existing Ed25519 key without duplicating key management.
- Add FRAMEWORK_MIGRATION.md documenting the 10 blockers surfaced
during a first-pass attempt (response-shape vs framework wrap,
McpServer CJS/ESM dual-resolution, VERSION_UNSUPPORTED /
dry_run / stateless-HTTP task-store edge cases, 30+ custom tools
outside AdcpToolMap, test-harness _requestHandlers path, ...) and
the staged 4-PR plan to unwind them.
No runtime behavior change — this commit just lays the groundwork.
The framework migration proper (replacing ~3,300 lines of hand-rolled
dispatch in task-handlers.ts) lands in a follow-up with dedicated
type-system budget.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds server/src/training-agent/framework-server.ts — createFrameworkTrainingAgentServer() routes all 30+ spec tools through @adcp/client/server's createAdcpServer via an adapt() helper that produces pre-formatted CallToolResult envelopes (preserving byte-identical responses), enforces VERSION_UNSUPPORTED, and maps exceptions to SERVICE_UNAVAILABLE per legacy behavior. The 9 tools outside AdcpToolMap (creative_approval, update_rights, comply_test_controller, validate_property_delivery, 5× *_collection_list) register directly via registerTool(). Resolver hooks configured: - idempotency: shared getIdempotencyStore() with legacy path - webhooks.signerKey: getWebhookSigningKey() (same Ed25519 key) - resolveIdempotencyPrincipal: scopedPrincipal() account-partitioning Opt in via TRAINING_AGENT_USE_FRAMEWORK=1; default OFF. The flag lets us land the scaffold without regression risk, diff response shapes per-tool under both paths, and flip the default in a focused follow-up PR. Handler exports in task-handlers.ts widened from module-local to export so framework-server.ts can import them. Legacy dispatch path is unchanged. Three gaps for the follow-up PR (tracked in FRAMEWORK_MIGRATION.md): 1. get_adcp_capabilities override — framework auto-registers this and registerTool() rejects duplicates. Training-agent-specific fields (publisher_domains, compliance_testing.scenarios, execution.targeting) need SDK support for replaceTool() or AdcpCapabilitiesConfig extension. 2. Webhook emission from dispatch — legacy path emits when push_notification_config.url is on the request; framework expects ctx.emitWebhook() from handlers. Need to either teach handlers to emit or keep a dispatch-level emitter on the framework path. 3. Stateless-HTTP task store verification — framework uses createTaskCapableServer; need empirical check under task-augmented create_media_buy to ensure notifications/tasks/status doesn't fail on a fresh transport per-request. Test status: 437/437 unit + integration tests green with flag OFF. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…/ customTools
5.4 closes the five upstream asks surfaced in 5.3:
- createAdcpServer() returns opaque AdcpServer (CJS/ESM hazard closed)
- AdcpServer.dispatchTestRequest({method,params}) replaces
_requestHandlers reach-through
- McpToolResponse.structuredContent is optional
- SingleAgentClient.validateRequest drops .strict() (no more
runner monkey-patch)
- Storyboard runner accepts request_signing.transport: 'mcp'
- New AdcpCustomToolConfig / config.customTools lets agents register
non-spec tools through the framework config (replaces
getSdkServer() escape hatch)
What changed here:
- framework-server.ts: return type any → AdcpServer. 9 non-AdcpToolMap
tools (creative_approval, update_rights, comply_test_controller,
validate_property_delivery, 5× *_collection_list) now register via
customTools: config instead of server.registerTool() after
construction. Zero escape hatches remaining on the type surface.
- run-storyboards.ts / run-one-storyboard.ts: drop the 33-line
SingleAgentClient.validateRequest monkey-patch. Pass
request_signing: { transport: 'mcp' } so signed_requests vectors
route through /mcp instead of per-op HTTP URLs.
Results: 437/437 unit + integration tests green (flag OFF). Storyboard
pass count unchanged (29/55 clean, 214 passing) — expected since the
framework path is still opt-in via TRAINING_AGENT_USE_FRAMEWORK=1.
signed_requests vectors now REACH /mcp (401 instead of 404 — they fail
because bearer auth runs before the signature verifier. Composing
bearer-or-signature via anyOf is a separate follow-up).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ia anyOf
5.5 adds verifySignatureAsAuthenticator so RFC 9421 signatures sit under
the same anyOf() composition as verifyApiKey — bearer OR signature is
sufficient, neither path short-circuits the other.
- request-signing.ts: Express middleware → Authenticator. getUrl override
uses req.originalUrl since Express strips the /api/training-agent mount.
- index.ts: signing authenticator joins the anyOf chain; legacy
requestSigningMiddleware wiring removed.
- framework-server.ts: capabilities.overrides declares publisher portfolio,
compliance_testing scenarios, and targeting surface for the framework
path (per-domain merge is 5.5-only).
- Runners: express.json({ verify }) captures rawBody bytes so the
in-process verifier rehashes exactly what the signer signed.
Framework flag stays opt-in — flipping it regresses storyboards due to
stricter zod arg validation that's not yet matched by our handler shapes.
29/55 storyboards clean on legacy path (+4 over 5.3 baseline). 437/437
unit + integration tests green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…olish Signed-requests composition: - anyOf(bearer, signature) accepted invalid signatures when a valid bearer was also present, breaking 25 negative conformance vectors. - New wrapper: if Signature-Input is present, signature MUST verify (throws propagate to 401). Otherwise bearer chain runs. - 401 responses embed the signing error code in a Signature WWW-Authenticate challenge (not Bearer) so the conformance grader can read it. - +19 signed_requests steps passing (9 → 28 of 34). - Upstream ask for spec-level helper: adcp-client#659. Review polish (from OAuth workstream review of #2561): - Real zod inputSchemas on all 9 custom tools (was _passthrough placeholder that published wrong contract via tools/list) — +34 framework-path steps passing. - Eager signing-authenticator init at boot so missing/corrupt compliance JWKS fail loud rather than as opaque 401. - Rename verifySignatureAsAuthenticator_ → lazySigningAuthenticator (the underscore shadowed the SDK export). - Idempotency scoping: comment why only static:public gets account- scoped; other principals use the auth principal directly. - useFrameworkServer jsdoc now matches code (default OFF). - Demote per-request framework-server-constructed log to debug. - Drop stale Stage 2 context comment. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Multiple storyboards share a brand domain (e.g. acme-outdoor used by governance_denied, sales_guaranteed, measurement_terms_rejected). The session store is keyed by brand domain, so a \$10K governance plan seeded by one storyboard persisted into the session and silently intercepted a \$50K buy in the next one — surfacing as GOVERNANCE_DENIED instead of the storyboard's expected outcome. clearSessions() before each storyboard isolates them. +3 storyboards clean (29 → 32), +13 steps passing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…e CI
- Storyboard runners serve /.well-known/oauth-protected-resource/<path>
at the top-level Express app (the probe hits \${origin}\${pathname},
not the mounted router path). +2 passing steps on security_baseline
(OAuth discovery phase).
- GitHub workflow runs the storyboard suite twice per PR: once with
TRAINING_AGENT_USE_FRAMEWORK=0 (legacy) and once with =1 (framework).
Each mode has a non-regression floor (clean storyboards + passing
steps) so CI fails loud if either path backslides. The framework
floor stays below the legacy one until zod parity lands and the flag
default flips.
32/55 clean, 256 steps passing (up from 25/54 baseline).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
5.6 ships the spec-compliant presence-gated auth helper we requested
(adcp-client#659). Replaces the local wrapper in buildAuthenticator
with the SDK version:
authenticate: requireSignatureWhenPresent(
lazySigningAuthenticator(),
anyOf(verifyApiKey(static), verifyApiKey(workos)),
)
The SDK also detects a solo Signature header (without Signature-Input)
which our naïve header check missed — closes one more signed_requests
negative vector that was silently accepting. Composition guard tag
(AUTH_PRESENCE_GATED) prevents accidental re-wrapping in anyOf.
Legacy: 33/55 clean, 261 steps (+1, +5).
Framework: 20/55 clean, 225 steps (+1, +9).
CI non-regression floors updated to match.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ble)
required_for enforcement on the unauthed fallback path:
- Capability now declares required_for: ['create_media_buy'].
- Pre-check in buildAuthenticator: when both signature and bearer are
absent, throw RequestSignatureError('request_signature_required') so
the conformance grader reads a Signature challenge with the right
error code. Bearer callers bypass — other storyboards unaffected.
SDK skipVectors for capability-profile mismatches:
- 007 (covers_content_digest: required vs our either)
- 018 (covers_content_digest: forbidden vs our either)
- 025 (grades SDK library verifier internals via jwks_override, not
our agent)
- skipRateAbuse: true for vector 020 (cap+1 requests; live side-effect
opt-in)
Legacy: 34/55 clean, 262 steps (+1, +1).
Framework: 21/55 clean, 226 steps (+1, +1).
signed_requests: 30/30 applicable vectors passing (was 29/34).
CI floors updated to match.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…nt-storyboard-pass # Conflicts: # package-lock.json # package.json # server/src/training-agent/framework-server.ts # server/src/training-agent/index.ts # server/src/training-agent/request-signing.ts # server/tests/manual/run-one-storyboard.ts # server/tests/manual/run-storyboards.ts
…cancel guard Seeding: - Products: test-product, sports_ctv_q2 aliased to real catalog entries (first-publisher / first-CTV-publisher). Pricing options test-pricing and default cloned onto test-product without min_spend_per_package so universal-suite \$1k budgets don't trip the min-spend gate. - Formats: video_30s, native_post, native_content added as explicit entries. - Buyer-supplied media_buy_id now honored on create_media_buy so storyboards that hardcode mb_acme_q2_2026_auction / mb_summer_campaign_001 can query by that literal. Spec feature wiring: - Past start_time rejection (schema_validation temporal_validation): reject with INVALID_REQUEST when start_time is >24h in the past. - Double-cancel guard (media_buy_state_machine terminal_enforcement): reject second cancel with INVALID_STATE_TRANSITION. 37/55 clean, 281 steps passing (was 34/55, 262). CI floors updated. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This was referenced Apr 20, 2026
Closed
Merged
…nt-storyboard-pass # Conflicts: # server/src/training-agent/index.ts # server/src/training-agent/request-signing.ts
…start revert
Four fixes for CI failures after merging main's two-route design:
1. FORMAT_CHANNEL_MAP missed video_30s / native_post / native_content —
the three format aliases I added in an earlier commit. Two unit tests
iterate buildFormats() and assert every id maps. Added mappings.
2. buildCatalog click_url unit test asserts every product's click_url
contains its product_id. My test-product / sports_ctv_q2 aliases
cloned the source product_card without updating the embedded URL.
Rewrite click_url on the cloned product_card{,_detailed} manifests
to substitute the alias id for the source id.
3. Past start_time rejection broke 5 status-derivation unit tests that
intentionally use 2020-dated flights to exercise derivation against
past dates. Tried narrowing to 180d — still not enough for the
"derives status from flight dates" test which uses 2020. Reverted;
schema_validation past_start step closes as accept-and-derive per
the storyboard's any_of branch on past_start_handled.
4. CI floor 37 → 35 (legacy) and passing-step floor 281 → 279 to match
the post-merge baseline. Framework floor's passing-step count also
updated 226 → 237 (gained from 5.6 + seed-fixture improvements in
main). Clean-storyboard floor stays at 21.
384/384 training-agent unit tests green. 35/55 storyboards clean with
279 steps passing.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #2561 that closes the highest-value gaps surfaced in the
OAuth workstream review and pushes the training agent from 25/54 → 32/55
clean storyboards (+7 clean, +72 passing steps). Work is split into
three commits so each is reviewable in isolation.
Depends on #2561 — this branch is cut from it to keep the 5.5 SDK
changes separate. Rebase to main once #2561 merges.
What landed
1. Presence-gated signature composition (25 → 6
signed_requestsfailures)anyOf(verifyApiKey, verifySignatureAsAuthenticator)had either/orsemantics: an invalid signature alongside a valid bearer would silently
accept the request, breaking every negative
signed_requestsconformance vector. Replaced with a wrapper: if
Signature-Inputispresent, the signature MUST verify; throws propagate to a 401 with
WWW-Authenticate: Signature error=\"<spec-code>\"so the grader canread the error code.
Upstream ask for a spec-level
requireSignatureWhenPresenthelperfiled as adcp-client#659.
2. Review polish (all 8 findings from pasted review)
inputSchemaon all 9 custom tools (was{ _passthrough: z.any().optional() }which published a wrongcontract via
tools/list). Closes 34 framework-path storyboardsteps blocked by the placeholder schema.
non-regression floors on clean-storyboards and passing-steps.
useFrameworkServerjsdoc now matches code (defaultOFF).
missing/corrupt compliance JWKS fail loud rather than as opaque 401.
verifySignatureAsAuthenticator_→lazySigningAuthenticator(was shadowing the SDK export of the samename).
static:publicis account-scoped.Stage 2comment.3. Storyboard isolation + OAuth discovery + CI
brand domain, so a governance plan seeded by one silently intercepted
buys in the next.
clearSessions()between storyboards → +3storyboards clean, +13 steps passing.
app (graders probe
\${origin}\${pathname}, not the mounted router).security_baseline: 2P/3F → 4P/1F.(legacy flag + framework flag) with regression floors per mode.
Results
TRAINING_AGENT_USE_FRAMEWORK=0)=1)What's still open (tracked for follow-up PR)
The remaining 23 failing storyboards break down as:
mb_acme_q2_2026_auction,campaign_hero_video,test-product,video_30s,sports_ctv_q2— storyboards reference these via hardcoded IDs thatdon't match the generated fixtures. Some are storyboard YAML drift
(should be capturing IDs from earlier steps); others need seed data.
INVALID_PARAMS / IDEMPOTENCY_CONFLICT error codes, past start_time
rejection,
vendor_costrequired,performance_index >= 0,double-cancel guard, approval
conditions, errors[] array inresponse, context echo on errors.
check_governanceneedscaller,build_creativepayload drift, etc. Unblocks frameworkflag default flip.
sync_creativesassets/name shape,
sync_catalogscatalog.type,sales_broadcast_tvidempotency_key regex. Separate upstream PR.required_for: [create_media_buy]configuration is incompatiblewith other storyboards running unsigned. Needs per-test-kit agent
configuration (spec-level discussion).
Test plan
npm run test:unit— 587/587 greennpm run typecheck— cleannpx tsx server/tests/manual/run-storyboards.ts— 32/55 clean, 256/57 stepsTRAINING_AGENT_USE_FRAMEWORK=1 npx tsx server/tests/manual/run-storyboards.ts— 19/55 clean, 216 steps🤖 Generated with Claude Code