docs(compliance): formalize storyboard runner semantics — fixtures, captures, context echo#2593
Conversation
Schema Link Check ResultsCommit:
|
Seven normative gaps surfaced by ad-tech-protocol-expert review of PR #2593: 1. Fixture shape — add non-normative pointer to domain schemas (product.json / creative.json / media-buy.json / plan schema) while keeping the fixture field additionalProperties: true. 2. Foreign-key DAG — complete the dependency graph. Previous edit only called out product→pricing_option and product→media_buy; add creative→media_buy, plan→media_buy, product→plan. Update both comply-test-controller.mdx and storyboard-schema.yaml. 3. Divergent-fixture tightening — second seed with diverging fixture MUST return INVALID_PARAMS, not MAY merge. Storyboards that need to mutate fixture state use force_* scenarios. Removes a storyboard determinism footgun. 4. Specialism-scoped seed floor — stateful compliance requires sellers to implement the seed_* scenarios covering the specialisms they certify against. Closes the "implement zero seeds to opt out" loophole while preserving UNKNOWN_SCENARIO → not_applicable for honest coverage reporting elsewhere. 5. Runner grading codes — enumerate capture_path_not_resolvable (producer), unresolved_substitution (consumer), fixture_seed_unsupported (not_applicable grading) so downstream tooling can discriminate them. 6. Webhook context echo cross-reference — add a "Echoing the caller's context object" subsection to webhooks.mdx pointing at the normative echo contract in context-sessions.mdx. Closes the gap where the new async-delivery echo rule was only surfaced in context-sessions. 7. Hardcoded-ID deprecation signal — explicitly state that existing hardcoded-literal fixtures are valid through 3.x and SHOULD migrate to the fixtures: block before 4.0. New storyboards MUST NOT hardcode. Also add the open-for-extension enum note (runners MUST accept unknown scenarios with UNKNOWN_SCENARIO rather than schema-reject). No wire-breaking changes. Review feedback only; all docs/schema tests still clean.
|
Reviewed against the three issues this closes (#2584 seed scenarios, #2585 fixtures/captures, #2589 context-echo). The DAG in the seed-ordering section (line 400) matches what we expected — Three findings worth addressing before merge. Blocker —
|
…aptures, context echo Closes #2584, #2585, #2589. 1. Adds `seed_*` scenarios to `comply_test_controller` (seed_product, seed_pricing_option, seed_creative, seed_plan, seed_media_buy). Storyboards can now declare prerequisite fixtures by stable ID without implementers guessing which IDs the conformance suite expects. 2. Adds declarative `fixtures:` block and `prerequisites.controller_seeding` flag to the storyboard schema. Runner auto-injects a fixtures phase that seeds via the new scenarios in foreign-key order. 3. Specs the existing `context_outputs:` + `$context.<name>` mechanism (already implemented in the runner, previously undocumented). 4. Tightens the context-echo contract — MUST echo on success AND error, MUST NOT synthesize when caller sent none, MUST NOT mutate. Runners MUST NOT auto-inject synthetic context. No wire-breaking changes. The comply_test_controller scenario enum is extensible; existing agents unaffected until they adopt seed_*.
Seven normative gaps surfaced by ad-tech-protocol-expert review of PR #2593: 1. Fixture shape — add non-normative pointer to domain schemas (product.json / creative.json / media-buy.json / plan schema) while keeping the fixture field additionalProperties: true. 2. Foreign-key DAG — complete the dependency graph. Previous edit only called out product→pricing_option and product→media_buy; add creative→media_buy, plan→media_buy, product→plan. Update both comply-test-controller.mdx and storyboard-schema.yaml. 3. Divergent-fixture tightening — second seed with diverging fixture MUST return INVALID_PARAMS, not MAY merge. Storyboards that need to mutate fixture state use force_* scenarios. Removes a storyboard determinism footgun. 4. Specialism-scoped seed floor — stateful compliance requires sellers to implement the seed_* scenarios covering the specialisms they certify against. Closes the "implement zero seeds to opt out" loophole while preserving UNKNOWN_SCENARIO → not_applicable for honest coverage reporting elsewhere. 5. Runner grading codes — enumerate capture_path_not_resolvable (producer), unresolved_substitution (consumer), fixture_seed_unsupported (not_applicable grading) so downstream tooling can discriminate them. 6. Webhook context echo cross-reference — add a "Echoing the caller's context object" subsection to webhooks.mdx pointing at the normative echo contract in context-sessions.mdx. Closes the gap where the new async-delivery echo rule was only surfaced in context-sessions. 7. Hardcoded-ID deprecation signal — explicitly state that existing hardcoded-literal fixtures are valid through 3.x and SHOULD migrate to the fixtures: block before 4.0. New storyboards MUST NOT hardcode. Also add the open-for-extension enum note (runners MUST accept unknown scenarios with UNKNOWN_SCENARIO rather than schema-reject). No wire-breaking changes. Review feedback only; all docs/schema tests still clean.
bokelley
force-pushed
the
bokelley/storyboard-runner-semantics
branch
from
2623007 to
4ca200b
Compare
- list_scenarios: MUST return canonical enum names verbatim. Custom scenario names aren't part of the compliance contract — the runner won't dispatch to non-enum scenarios, so listing them serves no purpose. - seed_pricing_option example: unify on floor_price (auction-based CPM, matches PricingOption schema) instead of the non-canonical 'rate' field. Add cross-reference to the schema so authors know which fields are legal. Review nits from PR author self-review. Not normatively changing any existing behavior.
|
Clarifying the "enum": ["INVALID_TRANSITION", "INVALID_STATE", "NOT_FOUND", "UNKNOWN_SCENARIO", "INVALID_PARAMS", "FORBIDDEN", "INTERNAL_ERROR"]This enum is distinct from the canonical seller-response Three canonical codes live in the spec today:
Ground truth:
So this PR is consistent with #2595 and #2596. The two nits ( |
check-schema-links CI caught that /schemas/v3/media-buy/product.json doesn't exist — the PricingOption schema lives at /schemas/v3/core/pricing-option.json. Also updated the prose reference in 'Fixture shape' to name the correct schema paths (core/product.json, core/pricing-option.json, core/media-buy.json, media-buy/sync-creatives-request.json).
|
Correction — retracting my earlier blocker on
The 7 PR #2596 is the one fixing the actual drift (the |
1 participant
Summary
Closes the three runner-semantics gaps surfaced by the training-agent conformance migration (#2582):
Fixture seeding (Spec: comply_test_controller fixture-seeding scenarios (seed_product, seed_pricing_option, seed_creative, seed_plan, seed_media_buy) #2584):
comply_test_controllergains fiveseed_*scenarios (seed_product,seed_pricing_option,seed_creative,seed_plan,seed_media_buy). Storyboards can now declare prerequisite fixtures by stable ID without implementers guessing which IDs the conformance suite expects.Declarative fixtures block (Spec: storyboard fixtures block / pattern — replace hardcoded fixture IDs #2585 Pattern A): new
fixtures:root block +prerequisites.controller_seeding: trueflag. Runner auto-injects a fixtures phase that seeds via the new scenarios in foreign-key order (products → pricing_options → media_buys).$context.<name>+context_outputs:(Spec: storyboard fixtures block / pattern — replace hardcoded fixture IDs #2585 Pattern B, Spec: context-echo semantics tightening (.* resolution and MUST-echo rules) #2589): specs the existing runner behavior — these mechanisms already work in the runner, but nothing in the spec documented them. Storyboard authors had to grep YAML to learn the substitution syntax.Context-echo contract tightening (Spec: context-echo semantics tightening (.* resolution and MUST-echo rules) #2589): MUST echo on success AND error, MUST NOT synthesize when caller sent none, MUST NOT mutate. Runners MUST NOT auto-inject synthetic
context:— storyboards whose validator checkscontext.correlation_idMUST declare the block on every sample_request.What changed
static/schemas/source/compliance/comply-test-controller-request.json— 5 new enum values, 5 newallOfbranches, newfixture/product_id/pricing_option_id/plan_idparams, examples.docs/building/implementation/comply-test-controller.mdx— 5 new scenario sections, seeding-semantics subsection (upsert, foreign-key ordering, sandbox scope,UNKNOWN_SCENARIO→not_applicable).static/compliance/source/universal/storyboard-schema.yaml— newfixtures:root block,prerequisites.controller_seeding, stepcontext_outputs:, context-accumulator + substitution section, context-echo contract section.docs/building/integration/context-sessions.mdx— new "Normative echo contract" subsection with six numbered rules (success echo, error echo, async updates, no synthesis, no mutation, no action).docs/contributing/storyboard-authoring.md— new "Fixtures and cross-step captures" section with Pattern A and Pattern B examples plus when-to-use-which table.Backwards compatibility
comply_test_controllerscenario enum is extensible;seed_*adds new values only. Agents that don't implement them returnUNKNOWN_SCENARIO, which the runner treats as a coverage gap (gradesnot_applicable, not failed). Existing agents are unaffected.fixtures:andprerequisites.controller_seedingare optional on storyboards. Existing storyboards that hardcode fixture IDs keep working.context_outputs:and\$context.<name>are specs for already-implemented runner behavior — no agent-side change required.Test plan
npm run test:schemas— 483 schemas validatenpm run test:examples— 31 example payloads validate against their schemas (includes the 2 new seed_* examples)npm run test:composed— 15 composed schema tests passnpm run test:unit— 627 tests passnpm run typecheckcleannpm run build:schemascleanly builds with the new enum valuesCloses #2584, #2585, #2589.
🤖 Generated with Claude Code