Skip to content

test(creative): add canonical supported formats storyboard#5147

Merged
bokelley merged 2 commits into
mainfrom
next-steps-v2
May 29, 2026
Merged

test(creative): add canonical supported formats storyboard#5147
bokelley merged 2 commits into
mainfrom
next-steps-v2

Conversation

@bokelley

Copy link
Copy Markdown
Contributor

Summary

Adds 3.1 creative-agent storyboard coverage for canonical creative.supported_formats build capabilities.

  • advertises implemented creative build capabilities with agent-local capability_id values
  • routes build_creative through the advertised canonical image capability and emits a canonical format_kind manifest
  • rejects unknown or unimplemented build targets with FORMAT_NOT_SUPPORTED
  • keeps 3.0 compatibility mode from accepting 3.1-only build capability selectors

Closes #4591.

Validation

  • expert review: protocol reviewer, code reviewer, docs reviewer
  • npm run typecheck
  • npx vitest run server/tests/unit/training-agent.test.ts --config server/vitest.config.ts
  • npm run test:storyboard-context-output-paths
  • npm run test:storyboard-sample-request-schema
  • npm run test:storyboard-validations-paths
  • npm run test:storyboard-response-schema
  • npm run build:compliance
  • npm run test:storyboards
  • npm run test:storyboards:3.0-compat
  • precommit hook: unit tests, dynamic import lint, callApi state-change lint, typecheck
  • prepush hook: version sync, current storyboard matrix, 3.0 compatibility storyboard matrix

@bokelley bokelley changed the title [codex] Add creative canonical supported formats storyboard test(creative): add canonical supported formats storyboard May 29, 2026
@bokelley bokelley marked this pull request as ready for review May 29, 2026 09:01

@aao-release-bot aao-release-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Follow-ups noted below. Right shape: advertise what you actually implement, fail-closed on unknown build targets, partition 3.0 from 3.1 at the resolver. The "training agent only ships one canonical (image) and rejects the rest" posture matches how a real 3.1 creative agent should bound its catalog.

Things I checked

  • ad-tech-protocol-expert: sound-with-caveats. capability_id reuse as target_format_id.id is conformant — format-id.json accepts the slug, agent owns its namespace, and creative-manifest.json:67-86 correctly partitions 3.1 (format_kind arm) from legacy (format_id arm). The PR's emitted { format_kind: 'image', assets } is the 3.1 arm exactly.
  • FORMAT_NOT_SUPPORTED error envelope is conformant — open code vocabulary, recovery: correctable in the taxonomy, details.accepted_values is the SHOULD-level canonical key, and per-format attribution via field: target_format_ids[${index}] (task-handlers.ts:193, :245) matches build-creative-response.json's mandate.
  • Fail-closed in 3.0 compat is correct: 3.0 buyers can't validate the format_kind manifest shape, so accepting training_image_generation as a forward-compat extension would silently bind them to a manifest they can't parse. Test at training-agent.test.ts:4376 confirms.
  • Changeset minor is correct for this surface. Old behavior — validFormatIds.get(unknown) → undefined → silent default 300x250 manifest — was lazy passthrough, not a documented contract. Narrowing it isn't a wire break.
  • Largest-file rule: new compliance YAML is 212 lines. Read end-to-end; scenario flow (discover → build → reject) is coherent and the expect_error: true / negative_path: payload_well_formed framing on the rejection phase is right.

Follow-ups (non-blocking — file as issues)

  • Mode 1 library-retrieval path now resolves creative.formatId through resolveTarget. task-handlers.ts:5969 (was unconditional validFormatIds.get). If a library creative was synced under a formatId that isn't in getFormats() and isn't aliased, build_creative({creative_id}) flips from "silent 300x250 manifest" to FORMAT_NOT_SUPPORTED. Probably fine for storyboard-seeded creatives, but worth either an inline assertion or a test pinning the library-creative path against a known-good formatId.
  • BUILD_CREATIVE_FORMAT_ALIASES['display_300x250_generative'] -> 'display_300x250' shadows an existing registered format at server/src/shared/formats.ts:521. Same dimensions, so today the shadow is invisible; but the registered generative format has a different asset shape (brief/logo/click_url). If anything reads format.assets off the resolved entry, the alias silently returns the wrong declaration. Either drop that one alias entry (registered format resolves on its own) or comment why dimension-passthrough must win.
  • Missing test for multi-target mid-batch rejection. Implementation correctly attributes per-format error via target_format_ids[${index}] (task-handlers.ts:193, :245), but no test in the new canonical creative build capabilities block exercises a mixed-good/bad batch. Worth one regression case asserting errors[0].field === 'target_format_ids[1]' for the bad slot.
  • Compliance scenario field_absent: creative.supported_formats[1].capability_id (canonical_supported_formats.yaml:463-465) locksteps the storyboard to "exactly one advertised capability." When a second canonical lands in SUPPORTED_CANONICAL_BUILD_CAPABILITIES, this storyboard breaks for unrelated reasons. The two [0] assertions above it (capability_id, format_kind, format_option_id absent) already cover the wire-shape check the storyboard's narrative claims. The [1] check is a training-agent fixture assertion living in a compliance scenario. Move it to the unit test (where the equivalent build_html5 absence check already lives at training-agent.test.ts:4341) or drop it.
  • canonical_catalog_version: '3.1' paired with one-entry supported_formats (task-handlers.ts:5342-5343). The schema description frames canonical_catalog_version as "runtime catalog the agent was built against" — typegen alignment. But a buyer reading { canonical_catalog_version: 3.1, supported_formats: [image] } will reasonably read "this agent implements 3.1 and only supports image builds." Pre-PR the field was paired with all canonical kinds; the asymmetry now is new. Worth confirming the intended semantic in build_creative.mdx.
  • Document target_format_id.id accepting capability_id slugs. ad-tech-protocol-expert flagged this is folklore — conformant but never written down. One sentence in docs/creative/task-reference/build_creative.mdx makes it normative instead of inferred.

Minor nits (non-blocking)

  1. buildManifest else-if is dead. task-handlers.ts:154-159 routes any target.formatKind !== 'image' to HTML assets under format_kind: target.formatKind. The only formatKind ever set is 'image' (SUPPORTED_CANONICAL_BUILD_CAPABILITIES has one entry). Either delete the branch and let the type guard force the next contributor to think about asset shape per canonical, or comment that it's a stub.
  2. buildCanonicalImageAssets hardcodes the acme-outdoor fixture — task-handlers.ts:84-102. headline: 'Trail-ready gear for every summit', landing_page_url: https://acmeoutdoor.example/trail-pro, image url: .../acme-outdoor/banner_300x250.jpg. Other build modes thread req.brand and req.creative_id into the HTML payload; the canonical image path is brand-locked. Either thread the brand through or rename to buildAcmeOutdoorImageFixture so the next reader knows it's a stub.
  3. Conventional-commit prefix. PR title is test(creative): …, but the change also narrows the advertised catalog, introduces alias-map resolution, and converts three build_creative modes from silent-passthrough to fail-closed. Notable choice — feat(creative) would have telegraphed the behavior change. Changeset itself is correctly minor, so the wire-versioning story is intact.

LGTM. Approving on the strength of the protocol-conformance verdict plus the fail-closed posture across the three build_creative modes.

@bokelley bokelley merged commit 4af7213 into main May 29, 2026
28 checks passed
@bokelley bokelley deleted the next-steps-v2 branch May 29, 2026 09:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Canonical-formats conformance storyboard track — gates 3.1 GA promotion rubric

1 participant