feat(storyboard): cross-step assertion registry + runner hooks (adcp#2639)

[bokelley]

Summary

Adds a cross-step assertion registry and onStart / onStep / onEnd lifecycle hooks to runStoryboard. Storyboards express per-step checks inline already (response_schema, field_value, http_status, …); this PR adds the missing layer for properties that must hold across a whole run — governance denial never mutates, idempotency dedup across replays, status transitions monotonic, context never echoes secrets on error.

Framework only. Concrete assertion modules (idempotency dedup, governance mutation-block, status monotonicity, secret redaction) live in adcontextprotocol/adcp alongside the specialisms that own them; they'll register via the new API and be declared on their specialism's invariants: [...] list.

What changed

• src/lib/testing/storyboard/assertions.ts — new module. registerAssertion(spec), getAssertion(id), listAssertions(), clearAssertionRegistry(), resolveAssertions(ids). Throws on duplicate ids at registration and on unknown ids at resolve.
• src/lib/testing/storyboard/types.ts — Storyboard gains optional invariants?: string[]. StoryboardResult gains optional assertions?: AssertionResult[]. New AssertionResult type (scope: "step" | "storyboard", optional step_id).
• src/lib/testing/storyboard/runner.ts — three insertions in executeStoryboardPass:
  1. Start: resolve storyboard.invariants → specs (fails fast on unknown id); init per-assertion state; await onStart(ctx) on each.
  2. Per-step: after each step completes, await onStep(ctx, stepResult); append each returned result into step.validations[] under check: "assertion" AND into the storyboard-level assertions[] with scope: "step"; flip result.passed = false on any failure so the existing count/stateful-cascade logic treats it as a validation failure.
  3. End: await onEnd(ctx); record results with scope: "storyboard"; factor assertionsFailed into overall_passed.
• Multi-pass aggregator concatenates per-pass assertions[] (no dedup — a divergence across passes should surface).
• Public exports from @adcp/client/testing for the registry + types.

Shape

registerAssertion({
  id: 'governance.denial_blocks_mutation',
  description: 'GOVERNANCE_DENIED responses must not acquire any resources',
  onStart(ctx) { ctx.state.acquired = []; },
  onStep(ctx, stepResult) {
    if (stepResult.response?.media_buy_id) ctx.state.acquired.push(stepResult.step_id);
    return [];
  },
  onEnd(ctx) {
    const denied = /* ...detect DENIED step in context... */;
    if (denied && ctx.state.acquired.length) {
      return [{ passed: false, description: 'resource acquired after denial', error: `steps: ${ctx.state.acquired.join(', ')}` }];
    }
    return [{ passed: true, description: 'no post-denial mutation' }];
  },
});

Storyboard wiring:

id: governance_spend_authority_sb
# ...
invariants:
  - governance.denial_blocks_mutation
  - idempotency.dedup
phases: [...]

Why gate overall_passed?

Assertions encode conformance properties — a run where every validation is green but plan_hash churned across re-renders, or a denied plan still acquired a buy, is not a conformant run. Treating assertions as advisory would recreate the problem storyboards already have (authors memorize scripts and ignore broader guarantees).

Test plan

• Unit: registry register/get/list/clear/resolve + duplicate and unknown-id errors.
• Integration: stub MCP agent + minimal storyboard; verify
  • onStart called once with fresh per-assertion state
  • onStep called for every step in order
  • Step-scoped failure surfaces in both step.validations[] and result.assertions[] and flips overall_passed
  • onEnd failure surfaces as scope: "storyboard" and flips overall_passed even when all steps pass
  • state carries across onStart → onStep → onEnd on the same assertion
  • Storyboards without invariants are unaffected (no assertions key on result; no hooks fire)
  • Unknown id in invariants throws at run start
  • Passing assertions don't regress overall_passed vs baseline
• Neighbor tests (storyboard-brand-invariant, storyboard-idempotency-invariant, storyboard-runner-contract) still green.
• Full suite: 39 pre-existing failures on this branch = 40 on unmodified main (delta is unrelated flake); no regressions attributable to this PR.

Non-goals

• No assertion implementations here — they ship from adcontextprotocol/adcp against specialism modules that can register at import time.
• No YAML-declarative assertion DSL. Programmatic TS registration is the contract; YAML just references ids.

Related

• Compliance: specialism-level invariants layer on storyboard runner adcp#2639 — origin issue (cross-step invariants on storyboards)
• Training agent: LLM red-team conformance runner (anti-teach-to-test) adcp#2630 — LLM red-team runner will reuse this registry as its oracle (same invariants, different path generator)
• Supersedes Training agent: buyer-side SDK smoke test (real DSP integration path) adcp#2624 (the fixed SDK smoke-test framing)

🤖 Generated with Claude Code