Skip to content

feat: composable scenario library for seller storyboards#528

Merged
bokelley merged 9 commits into
mainfrom
bokelley/storyboard-refine-gap
Apr 15, 2026
Merged

feat: composable scenario library for seller storyboards#528
bokelley merged 9 commits into
mainfrom
bokelley/storyboard-refine-gap

Conversation

@bokelley

@bokelley bokelley commented Apr 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • Introduces a composable scenario library for media buy seller certification
  • Scenarios are small, focused behavior tests in storyboards/scenarios/media_buy_seller/
  • Specialization storyboards declare requires_scenarios — the compliance engine resolves and runs them alongside the main storyboard
  • Governance scenarios use real plans on the training agent (multi-agent testing)
  • Fixes signal_marketplace storyboard to chain discovered IDs instead of hardcoded ones

Motivation

Specialization storyboards (broadcast, non-guaranteed, etc.) skipped shared behaviors like governance integration, product refinement, and delivery reporting. Sellers could pass certification without proving they handle governance denial, product refinement, or schema-compliant delivery responses.

Scenarios shipped

Scenario Tests Status vs training agent
governance_approved Plan allows $100K, $25K buy passes 5/5 pass
governance_denied Plan allows $10K, $50K buy denied Blocked on adcontextprotocol/adcp#2185
governance_conditions Custom policies, CTV buy approved with conditions 5/5 pass
refine_products Product-level refinement with refinement_applied 3/3 pass
proposal_finalize Draft -> refine -> finalize -> committed -> IO 5/5 pass
delivery_reporting simulate_delivery -> get_media_buy_delivery -> schema check Blocked on training agent task error propagation

Storyboard requirements

Storyboard Governance Refine Finalize Delivery
media_buy_seller Yes Yes - Yes
media_buy_proposal_mode Yes - Yes Yes
media_buy_guaranteed_approval Yes Yes - Yes
media_buy_broadcast_seller Yes Yes - Yes
media_buy_catalog_creative Yes Yes - Yes
media_buy_generative_seller Yes Yes - Yes
media_buy_non_guaranteed Yes - - Yes

TypeScript changes

  • types.ts — requires_scenarios field on Storyboard interface
  • loader.ts — recursive scenario loading from storyboards/scenarios/, getScenarioById(), resolveRequiredScenarios()
  • comply.ts — expandScenarios() with dedup, nested scenario rejection, throw on missing IDs
  • schema.yaml — documented requires_scenarios

Additional fix

signal_marketplace.yaml — steps after search_by_spec now use context_outputs from discovery instead of hardcoded signal IDs that may not exist on the agent being tested.

Follow-up issues

Test plan

  • npm run build passes
  • npm test passes (2832/2832)
  • npx adcp storyboard list — scenarios don't appear in listing
  • Scenario expansion deduplicates correctly
  • expandScenarios throws on missing scenario IDs (not silent skip)
  • expandScenarios rejects nested requires_scenarios
  • governance_approved: 5/5 pass against training agent
  • governance_conditions: 5/5 pass against training agent
  • refine_products: 3/3 pass against training agent
  • proposal_finalize: 5/5 pass against training agent
  • CI passes

🤖 Generated with Claude Code

bokelley and others added 2 commits April 15, 2026 05:28
@bokelley @claude
feat: add composable scenario library for seller storyboard certifica…
7755857 
…tion

Decompose the base media_buy_seller storyboard into reusable scenarios
that specialization storyboards compose via requires_scenarios. Scenarios
are small, focused behavior tests organized by agent type under
storyboards/scenarios/media_buy_seller/.

Initial scenarios:
- governance_approved/denied/conditions (using force_governance_outcome)
- refine_products (product-level refinement)
- proposal_finalize (draft -> refine -> finalize -> committed -> IO)

All 7 media buy seller storyboards now declare requires_scenarios.
The compliance engine resolves and runs them alongside the main storyboard.

Closes #527 partially (governance + refine scenarios shipped; creative,
async, and error scenarios tracked in the issue for follow-up).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@bokelley @claude
style: fix prettier formatting
8c21983 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@bokelley bokelley mentioned this pull request Apr 15, 2026
Closed
bokelley and others added 2 commits April 15, 2026 05:42
@bokelley @claude
fix: remove force_governance_outcome, governance scenarios need multi…
2666054 
…-agent harness

Governance is an external call the seller makes to a governance agent —
the right way to test it is to control what's on the other end of that
call, not to tell the seller to pretend it got a specific answer.

Updated governance scenarios to document that the test harness must act
as the governance agent. Added $harness.governance_url placeholders.
Removed force_governance_outcome from the test controller.

See #529 for the multi-agent harness capability needed to run these.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@bokelley @claude
refactor: governance scenarios use real plans instead of forced outcomes
b1e88ca 
Governance is a multi-agent interaction — the test harness sets up real
governance plans (spending limits, custom policies) on a governance agent,
then registers that agent with the seller. The seller calls check_governance
during create_media_buy and gets a real response.

The training agent acts as the default governance agent. Override with
--governance-agent-url for custom/local governance agents.

Scenarios:
- governance_approved: permissive plan ($100K limit), $25K buy passes
- governance_denied: strict plan ($10K limit), $50K buy denied
- governance_conditions: conditional plan (CTV reporting), CTV buy approved with conditions

See #529 for remaining harness integration work.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This was referenced Apr 15, 2026
Closed
Closed
bokelley and others added 5 commits April 15, 2026 21:53
@bokelley @claude
feat: add delivery_reporting scenario to all seller storyboards
ce8daca 
Creates media_buy_seller/delivery_reporting scenario that chains:
simulate_delivery (via comply_test_controller) → get_media_buy_delivery
→ schema validation. Catches sellers returning non-compliant delivery
response formats.

All 7 media buy seller storyboards now require this scenario.

Closes #543.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@bokelley @claude
fix: signal_marketplace storyboard uses discovered IDs instead of har…
9043421 
…dcoded ones

Steps after search_by_spec now use context from the discovery step
(signal_id, signal_agent_segment_id, pricing_option_id) instead of
hardcoded IDs that may not exist on the agent being tested.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@bokelley @claude
fix: include final governance scenario rewrites
7f36ac0 
These files were rewritten with the real-plan approach but the final
versions weren't included in the earlier commit.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@bokelley @claude
chore: merge main, resolve conflicts with context/correlation_id addi…
517921f 
…tions

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@bokelley
Merge remote-tracking branch 'origin/main' into bokelley/storyboard-r…
e1568fa 
…efine-gap
@bokelley bokelley merged commit d99b118 into main Apr 15, 2026
11 checks passed
@bokelley bokelley deleted the bokelley/storyboard-refine-gap branch April 15, 2026 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@bokelley