Skip to content

feat(compliance): cover vendor_metric optimization goals#4972

Merged
bokelley merged 5 commits into
mainfrom
bokelley/vendor-metric-coverage
May 24, 2026
Merged

feat(compliance): cover vendor_metric optimization goals#4972
bokelley merged 5 commits into
mainfrom
bokelley/vendor-metric-coverage

Conversation

@bokelley

@bokelley bokelley commented May 23, 2026

Copy link
Copy Markdown
Contributor

Closes #4933

Summary

Adds compliance coverage for the AdCP 3.1 optimization_goals[].kind: "vendor_metric" contract. The new media-buy storyboard verifies discovery of vendor_metric_optimization.supported_metrics[], positive create-media-buy acceptance when capability and reporting-coherence preconditions are met, and negative paths for unsupported vendor metrics, unsupported target kinds, and missing vendor-scope committed_metrics[].

The training agent now publishes vendor-metric optimization capability on both capability surfaces, ranks vendor-metric-capable products higher for relevant briefs, and enforces that vendor-metric goals are both optimizable and reportable before accepting a media buy.

Expert Review

Ran protocol, ad-tech protocol, and code-review expert passes. Follow-up fixes addressed their findings:

  • TERMS_REJECTED is used for vendor-metric capability/reporting-coherence rejection paths.
  • The storyboard is gated on seller support for media_buy.vendor_metric_optimization.supported_targets containing threshold_rate.
  • Unsupported metric and unsupported target negative paths are isolated and assert errors[0].field.
  • Runtime validation now rejects metrics that are optimizable but missing from reporting_capabilities.vendor_metrics.
  • Brief discovery now reliably surfaces vendor-metric optimization products for attention/vendor-metric briefs.

Validation

  • npm run test:storyboard-validations-paths
  • direct sample-request and context-output lint on vendor_metric_optimization_flow.yaml
  • npm run build:compliance
  • npx vitest run server/tests/unit/training-agent.test.ts --config server/vitest.config.ts
  • npm run typecheck
  • npm run test:storyboards
  • npm run test:storyboards:3.0-compat
  • precommit hook: test:unit, test:test-dynamic-imports, test:callapi-state-change, typecheck
  • push hook: version sync, current storyboard matrix, 3.0 storyboard compatibility matrix

Known unrelated repo drift remains in the full global lints: test:storyboard-sample-request-schema reports protocols/creative/scenarios/native_in_feed.yaml, and test:storyboard-context-output-paths reports specialisms/sales-guaranteed/index.yaml.

@bokelley
bokelley marked this pull request as ready for review May 23, 2026 21:19
aao-release-bot[bot]
aao-release-bot Bot previously approved these changes May 23, 2026

@aao-release-bot aao-release-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Follow-ups noted below. Clean encoding of the AdCP 3.1 vendor_metric contract — capability, target-kind, and reporting-coherence checks mirror the schema's normative wording, and the storyboard's INVALID_REQUEST vs TERMS_REJECTED framing matches the error-code scoping.

Things I checked

  • Schema-vs-implementation coherence: the three preconditions in static/schemas/source/core/optimization-goal.json:183 map 1:1 onto the validator at server/src/training-agent/task-handlers.ts:1998-2068. Capability key uses (vendor.domain, brand_id, metric_id) — symmetric with vendor_metric_values, reporting_capabilities.vendor_metrics, and committed_metrics.
  • Validator does not fire on unrelated goal kinds — early returns at L2021 (goal.kind !== 'vendor_metric') and L2023 (!key). Sits after productMap build, mirrors the existing metric_optimization validator's shape directly above.
  • Storyboard fixture (static/compliance/source/protocols/media-buy/scenarios/vendor_metric_optimization_flow.yaml:79-85) matches vendor-metric-optimization.json required fields; sample requests correctly omit committed_at (response-stamped per package.json:74).
  • Test helper at server/tests/unit/training-agent.test.ts:104-105 unwraps errors[0] into result, so result.code / result.field assertions in the four new tests read the right path.
  • Empty-frontmatter changeset matches established repo precedent (.changeset/4321-prospect-on-conflict-do-nothing.md, .changeset/4348-pipes-github-connection-state-lag.md) — non-package-bumping changes.
  • ad-tech-protocol-expert: sound-with-caveats — three preconditions match optimization-goal.json:183, INVALID_REQUEST correct per error-code.json:90 scoping, no TERMS_REJECTED conflict.
  • code-reviewer: no blockers; control flow consistent with the existing metric_optimization validator pattern.

Follow-ups (non-blocking — file as issues)

  • Storyboard skips the SHOULD-level discovery precondition. The narrative at L26-31 names three preconditions but only two are exercised. The third (metric_id SHOULD appear in vendor's measurement.metrics[]) tightens to MUST at the next minor — worth a forward-compat phase now so sellers that hard-fail on catalog miss get flagged at the 3.2 boundary.
  • No target-less positive case. Validator at L2038-L2052 correctly bypasses supported_targets when target is omitted (target-less = maximize within budget), but the storyboard never exercises that branch. Add a positive phase that omits target so implementations get coverage on the "optional target" semantics.
  • vendorMetricKey lowercases domain but not brand_id (task-handlers.ts:170-178). Brand-ref schema is silent on brand_id case; case-equal would be the safer reconciliation rule and matches vendor_metric_values symmetry. Non-blocking — current behavior is defensible.

Minor nits (non-blocking)

  1. Unknown product_id masked as capability miss. task-handlers.ts:2012-2014 — when pkg.product_id resolves to no product, product is undefined and the goal is rejected as "not in supported_metrics" instead of the later PRODUCT_NOT_FOUND. Matches the existing metric_optimization validator's precedent at L1953/L1963, so it's consistent, not a regression — flagging only because a comment naming the masking would prevent the next reviewer from re-discovering it.
  2. Target validated only when kind is a string. task-handlers.ts:2038-2052target: {} or target: null silently bypasses the supported_targets check. Schema validation should catch malformed shapes earlier, but the metric-kind validators above probe types more defensively. Cheap to tighten.
  3. Negative-test fragility on bid_price floors. findProductWithVendorMetric (training-agent.test.ts:1893-1908) picks the first matching product and the first pricing option. If publisher fixture reordering ever pushes a fixed-floor or min_spend_per_package pricing variant to position 0, the bid_price=5.0 / budget=10000 in the three negative tests could trip a different validator first. Tests assert on result.field so the regression would surface, but the helper should prefer a known-floor template the way findProductWithMetric callers already navigate.

Approving on the strength of schema-conformant precondition logic plus the storyboard's clean error-envelope framing. The two unexercised paths (discovery SHOULD, target-less positive) are worth a follow-up issue but not load-bearing for shipping the contract test.

@aao-release-bot aao-release-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly clean. Holding for a question on the seller-level capability gate path and an error-code consistency question.

Things I checked

  • task-handlers.ts:2026-2117 — vendor_metric validation block. Order of checks (capability → target → committed → reportable) is correct. vendorMetricKey brand_id strictness matches static/schemas/source/core/vendor-metric-optimization.json uniqueness key. Reportability check at :2096-2098 correctly returns false when reporting_capabilities.vendor_metrics is undefined. Every dereference is optional-chained.
  • task-handlers.ts:3402-3405 — seller-level rollup vendor_metric_optimization: { supported_targets: ['threshold_rate'] }. The media_buy capabilities block in static/schemas/source/protocol/get-adcp-capabilities-response.json is additionalProperties: true (line 665), so this is permitted — but the field is not defined in the schema. Sibling rollup supported_optimization_metrics (line 547) is the established pattern.
  • training-agent.test.ts:1890-2263 — five tests cover five distinct failure paths. Each asserts a distinct errors[0].field JSONPath. No overlap that would mask a regression.
  • vendor_metric_optimization_flow.yaml — committed_metrics shape (scope: vendor, vendor, metric_id) matches media-buy/package-request.json:151-169. JSONPath-lite field assertions conform to core/error.json#field. Changeset present (empty-frontmatter pattern matches 2487-image-size-cap-for-c2pa-manifest.md and friends — established convention for build/test-only changes).
  • PR description test plan — all change-path commands listed without unchecked items. The two reported "unrelated repo drift" items are pre-existing and not on the path this PR changes.

Open questions (would flip to approve)

  1. Seller-level capability gate. The storyboard's requires_capability.path is media_buy.vendor_metric_optimization.supported_targets. That path is not currently defined in static/schemas/source/protocol/get-adcp-capabilities-response.json — only supported_optimization_metrics exists at the seller level. The agent emits the extension field at task-handlers.ts:3402, and additionalProperties: true permits it. Two paths forward:

    • Land a parallel schema PR adding vendor_metric_optimization (with supported_targets) to the media_buy capabilities block — so any seller, not just this training agent, knows what to publish. The storyboard then gates on a real schema-defined field.
    • Or gate on the product-level path that IS in the schema (vendor_metric_optimization.supported_metrics[].supported_targets per core/vendor-metric-optimization.json:23-30), and drop the seller-level emission.

    As written, the gate works for this agent because the agent publishes the extension, but no other conformant seller has a wire-defined signal to publish that would make them match the gate — they'd grade not_applicable forever. Is a schema PR queued?

  2. Error-code consistency with sibling capability checks. The new code at task-handlers.ts:2060 and :2076 returns TERMS_REJECTED for "goal value not in declared product capability." The sibling checks for the same shape — reach_unit not in supported_reach_units at :1999, view_duration_seconds not in supported_view_durations at :2015 — return INVALID_REQUEST. Per enums/error-code.json:138, TERMS_REJECTED is scoped to measurement_terms rejection. The reporting-coherence checks at :2086 and :2099 are defensibly TERMS_REJECTED; the two capability checks above them read more like INVALID_REQUEST to match sibling precedent. Is the uniform TERMS_REJECTED choice deliberate?

Follow-ups (non-blocking — file as issues)

  • Brief-relevance heuristic at task-handlers.ts:1485-1488 matches only "attention" or "vendor" substrings. core/vendor-metric-optimization.json:5 calls out emissions and brand-lift as core use cases — sustainability briefs ("emissions", "scope3") and lift briefs ("brand lift") miss the vendor-metric bonus. Tightening to those keywords generalizes discovery for non-attention vendor metrics.
  • Storyboard narrative parity with unit tests. The fourth runtime check at task-handlers.ts:2096-2107 ("optimizable but not reportable") has unit coverage at training-agent.test.ts:2184 but no corresponding negative phase in the storyboard. Either add a 4th negative phase, or note that the committed_metrics schema (media-buy/package-request.json:107) may already constrain this and the runtime check is belt-and-suspenders.
  • vendor_metric_optimization applies to guaranteed and non-guaranteed products at product-factory.ts:565, unlike metric_optimization which gates on non-guaranteed (:406). Probably fine but worth a one-line decision.

Minor nits (non-blocking)

  1. Handler error message at task-handlers.ts:2076 says "vendor_metric_optimization.supported_targets" — the true per-product path is vendor_metric_optimization.supported_metrics[i].supported_targets. The shortened path is fine in user-facing prose but reinforces the seller-level-rollup framing flagged in question 1 above.

Notable choice on TERMS_REJECTED for capability mismatches — worth confirming before this code becomes the precedent the next vendor-extension PR copies from.

@bokelley

Copy link
Copy Markdown
Contributor Author

Argus follow-up addressed in f3ad05053e.

Open questions:

  1. Seller-level capability gate: resolved in this PR. I added a schema-defined media_buy.vendor_metric_optimization.supported_targets object to static/schemas/source/protocol/get-adcp-capabilities-response.json, so the storyboard gate is no longer relying on an extension-only field. Product-level vendor_metric_optimization.supported_metrics[] remains authoritative for the actual (vendor, metric_id) pairs.

  2. TERMS_REJECTED for vendor-metric capability misses: keeping this deliberately. The sibling seller-native metric checks still use INVALID_REQUEST, but vendor-metric optimization is negotiated against vendor measurement/reporting terms. The normative docs already say vendor-metric capability and reporting-coherence misses reject with TERMS_REJECTED (optimization-reporting.mdx and conversion-tracking/index.mdx). I added an inline handler comment so future reviewers do not infer this was accidental drift from the seller-native metric validators.

Follow-ups/nits addressed:

  • Added target-less positive storyboard coverage for maximize-within-budget semantics.
  • Added storyboard coverage for optimizable-but-not-reportable vendor metrics.
  • Generalized the brief-relevance heuristic beyond attention/vendor to emissions, sustainability/scope3/carbon, brand-lift/awareness/incremental/panel language.
  • Added a product-factory comment documenting why vendor_metric_optimization is not limited to non-guaranteed products.
  • Fixed the target-kind rejection message to point at vendor_metric_optimization.supported_metrics[].supported_targets.
  • Tightened malformed target handling (target present without kind now returns INVALID_REQUEST on ...target.kind).
  • Added the unknown-product masking comment requested in the nit.
  • Made findProductWithVendorMetric prefer a pricing option that will not trip floor/min-spend validators before the vendor-metric assertion.

Split out as follow-up issue:

One item intentionally unchanged from the earlier dismissed review: brand_id remains case-sensitive in vendorMetricKey; the schema describes the semantic key as (vendor.domain, vendor.brand_id, metric_id), and brand IDs are identifiers rather than domains.

@aao-release-bot aao-release-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes on the changeset. Schema-source change without a versioned changeset ships the new seller capability untracked at the protocol version surface.

Blocking

  1. .changeset/vendor-metric-optimization-storyboard.md has empty frontmatter on a schema-source-touching PR. The diff edits static/schemas/source/protocol/get-adcp-capabilities-response.json (lines 569-588 — new seller-level vendor_metric_optimization block). Every prior PR that added an analogous seller-level capability rollup used "adcontextprotocol": minor.changeset/4651-supported-optimization-metrics-capability.md and .changeset/4639-conversion-tracking-supported-targets.md. The empty-frontmatter precedent cited in vendor-metric-accountability-storyboard.md was storyboard-only with no schema-source edit, so it doesn't carry. Fix: set frontmatter to "adcontextprotocol": minor. The schema addition is additive (optional field, additionalProperties: true), so minor is the right type — not major.

Things I checked

  • Validator control flow at server/src/training-agent/task-handlers.ts:2021-2125 — nested loops index correctly, each failure short-circuits the request, ordering mirrors the sibling metric_optimization block and matches the per-step negative-path expectations in the storyboard.
  • vendorMetricKey canonicalization (task-handlers.ts:159-182) — lowercases domain only; metric_id and brand_id are schema-pinned lowercase via vendor-metric-id.json / brand.json, so the asymmetry is correct.
  • Schema enum parity — seller-level supported_targets enum ["cost_per","threshold_rate"] matches product-level static/schemas/source/core/vendor-metric-optimization.json:29 exactly.
  • Error-code choice — TERMS_REJECTED for capability and reporting-coherence misses tracks docs/media-buy/media-buys/optimization-reporting.mdx:79-81; INVALID_REQUEST for malformed target.kind follows the structural-validation convention. Right shape.
  • Test honesty — five negative-path tests plus two positive paths, all assert errors[0].field and code. PR description's validation block is fully checked; nothing claimed-but-unverified.
  • 3.0-compat — additive optional field with additionalProperties: true, no risk to 3.0 storyboard matrix.

Follow-ups (non-blocking — file as issues)

  • Storyboard gate over-narrows. requires_capability.contains: "threshold_rate" at vendor_metric_optimization_flow.yaml:13-15 marks cost_per-only sellers not_applicable for the entire scenario, including the target-less positive path and three of the four negative paths that don't depend on threshold_rate. Either split the scenario or gate only the unsupported-target-kind step on threshold_rate.
  • Seller-level surface omits supported_metrics rollup. Buyers can't discover which (vendor, metric_id) pairs any product supports without walking the catalog. additionalProperties: true keeps the door open; track for a follow-up minor.

Minor nits (non-blocking)

  1. Misleading brief_relevance string at task-handlers.ts:1517. When channelScore === 0 && keywordScore === 0 but vendorMetricScore > 0, the response reads "Matches keywords only with vendor-metric optimization." — there were no keyword matches. Render as "Matches vendor-metric optimization." in that branch.
  2. findProductWithVendorMetric() at training-agent.test.ts:1893-1913 depends on a single publisher. Pinnacle (publishers.ts:36-44) is the only profile carrying vendor_metric_optimization. If that fixture loses or renames attention_score, every vendor-metric test throws. The seeded-fixture path via seedVendorMetricProduct already exists — consider routing the capability-positive tests through it too.
  3. VENDOR_METRIC_OPTIMIZATION_BRIEF_TERMS ranking widens brief results to vendor-metric products even with zero channel/keyword score. Intentional per the discovery test, but worth a one-line doc comment so future ranking edits don't strip it.

Ship it once the changeset frontmatter is set to "adcontextprotocol": minor.

@bokelley

Copy link
Copy Markdown
Contributor Author

Argus follow-up pass is pushed in e243e88.

Addressed:

  • Changeset is now a minor bump because the schema source changed.
  • Storyboard gating now only requires seller-level vendor_metric_optimization presence; the product-level seeded fixture remains authoritative for target support, so cost_per-only sellers won't skip the whole scenario because threshold_rate is absent.
  • brief_relevance now describes the actual match sources, so vendor-metric-only matches no longer read as keyword-only matches.
  • Added the one-line doc comment for the widened vendor-metric ranking intent.

Follow-ups tracked / intentionally left:

  • Seller-level supported_metrics rollup is now tracked as Add seller-level supported_metrics rollup for vendor_metric_optimization #4988.
  • I left findProductWithVendorMetric catalog-bound for this PR: the storyboard fixture is the contract path under test here, and the existing seeded/overlay tests cover the separate custom-product path. Happy to change that if you want it in this PR, but I treated it as non-blocking.

Validation:

  • npm run build:compliance
  • npx vitest run server/tests/unit/training-agent.test.ts --config server/vitest.config.ts
  • npm run typecheck
  • pre-push current-source storyboard matrix
  • pre-push 3.0 compatibility storyboard matrix

@bokelley

Copy link
Copy Markdown
Contributor Author

The findProductWithVendorMetric catalog-bound call is sound — the storyboard fixture is the right contract surface under test here, and the existing seeded/overlay tests covering the custom-product path mean there's no coverage gap. Expanding scope to the custom-product path would be nice-to-have, not blocking.

Seller-level supported_metrics rollup tracked in #4988 is the right home for that work.


Triaged by Claude Code. Session: https://claude.ai/code/${CLAUDE_CODE_REMOTE_SESSION_ID}


Generated by Claude Code

@bokelley

Copy link
Copy Markdown
Contributor Author

Issue #4988 proposes adding a seller-level supported_metrics rollup to vendor_metric_optimization — same surface as this PR. Consider folding before merge or confirm as a tracked follow-up.


Generated by Claude Code

@bokelley

Copy link
Copy Markdown
Contributor Author

Confirmed as tracked follow-up — folding #4988 into this PR is not the right call.

Three reasons:

  1. Design question, not code-ready. Add seller-level supported_metrics rollup for vendor_metric_optimization #4988's first acceptance criterion is "Decide whether seller-level vendor_metric_optimization should include supported_metrics" — the schema shape and contract aren't settled. Adding it here would introduce open-ended scope to an otherwise well-defined PR.
  2. PR is large and non-draft. At 1,111 additions across 9 files, scope expansion for schema definition + docs + compliance coverage would materially delay review.
  3. Intentionally scoped out. The issue body says "Follow-up from PR feat(compliance): cover vendor_metric optimization goals #4972 and Argus review" — this was deliberately filed as a follow-up, not an oversight.

#4988 stays open with claude-triaged. The triage comment on it carries the Blocked-on: #4972 signal so it resurfaces automatically after this PR merges.


Generated by Claude Code

aao-release-bot[bot]
aao-release-bot Bot previously approved these changes May 24, 2026

@aao-release-bot aao-release-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Follow-ups noted below. Validator preconditions, storyboard assertions, and the seller-level rollup line up cleanly — capability → target shape → target support → committed presence → reportability, in that order, with field paths that match what tests and YAML actually assert.

Things I checked

  • Validator order and field paths at server/src/training-agent/task-handlers.ts ~L2030-2140 match the storyboard's errors[0].field assertions one-for-one (packages[0].optimization_goals[0].metric_id at vendor_metric_optimization_flow.yaml:380, …target.kind at :440, packages[0].committed_metrics at :495 and :556) and the unit-test expectations at server/tests/unit/training-agent.test.ts:2188, :2217, :2299, :2326.
  • vendorMetricKey at task-handlers.ts:175-184 normalizes domain casing on both sides of the comparison and treats missing brand_id as '' consistently, so a vendor: { domain }-only goal matches a vendor: { domain }-only supported entry. No false-negative path on this key.
  • Seller-level rollup parity: handleGetAdcpCapabilities at task-handlers.ts:3453 returns vendor_metric_optimization: { supported_targets: ['threshold_rate'] }; v6 platform at v6-sales-platform.ts:191 declares the same; schema enum at get-adcp-capabilities-response.json:577-586 permits exactly cost_per and threshold_rate. On the wire and in code they agree.
  • Schema change is purely additive optional with additionalProperties: true mirroring sibling supported_optimization_metrics at get-adcp-capabilities-response.json:547-568. minor changeset is the right shape — non-breaking capability surface.
  • Publisher fixture at publishers.ts:33-43 adds attention_score to both vendorMetrics and vendorMetricOptimization.supported_metrics[] — has to land in both for the validator's two-sided coherence check to clear, and it does.
  • TERMS_REJECTED for capability/coherence rejections is normatively right per docs/media-buy/media-buys/optimization-reporting.mdx:79-81, vendor-metric-optimization.json:5, and optimization-goal.json:183 — distinct from the INVALID_REQUEST used by sibling reach/cpcv shape-check flows, which is the intended distinction (terms vs. shape).
  • Six new unit tests assert code, field, and message substring per path — not just smoke-tested happy paths.

Follow-ups (non-blocking — file as issues)

  • error-code.json:138 still describes TERMS_REJECTED narrowly as 'Buyer-proposed measurement_terms were rejected.' This PR's normative basis for using it on vendor-metric optimization-goal preconditions is sound, but an implementor reading just the enum description will reasonably pick INVALID_REQUEST. Broaden the enum description to cover vendor-metric optimization-goal rejections.
  • Pre-existing docs drift on docs/protocol/get_adcp_capabilities.mdx: the page documents neither supported_optimization_metrics nor the new vendor_metric_optimization rollup. Not caused by this PR but amplified by it.
  • Malformed vendor_metric goal with missing vendor.domain or metric_id continues silently via the if (!key) continue guard at task-handlers.ts:2089. If no downstream validator catches it, the buy lands. Consider returning INVALID_REQUEST with field: …optimization_goals[j].vendor / .metric_id for symmetry with the target-shape branch.

Minor nits (non-blocking)

  1. brief_relevance reads awkward at three matches. With channel+keyword+vendor-metric all positive, the joined string at task-handlers.ts:1527 produces Matches 1 channel(s) and keywords and vendor-metric optimization. Two ands. Oxford-list join would tighten it.
  2. vendorMetricKey doesn't trim domain. task-handlers.ts:181 lowercases but doesn't .trim(). No current fixture has whitespace; cheap insurance against fixture drift.

LGTM. Follow-ups noted below.

@bokelley

Copy link
Copy Markdown
Contributor Author

Additional expert-review fixes pushed in 6a81b34.

Addressed before merge:

  • Unknown product_id with a vendor_metric goal now stays on the normal PRODUCT_NOT_FOUND path instead of being masked as TERMS_REJECTED.
  • Malformed vendor_metric goals now reject structurally when vendor.domain or metric_id is missing/empty.
  • vendor_metric target.value is now required to be a positive finite number whenever target is present.
  • Brief ranking now scores the best matching supported vendor metric rather than summing all supported_metrics rows, and derives outcome category matches from metric identity so generic emissions briefs prefer emissions products.
  • Added direct legacy capability coverage and v6 parity coverage for seller-level vendor_metric_optimization.supported_targets.

Validation:

  • npx vitest run server/tests/unit/training-agent.test.ts -t "vendor_metric|get_adcp_capabilities" --config server/vitest.config.ts
  • npx vitest run server/tests/unit/training-agent.test.ts --config server/vitest.config.ts
  • npm run build:compliance
  • npm run build:schemas
  • pre-push current-source storyboard matrix
  • pre-push 3.0 compatibility storyboard matrix

Note: local npm run typecheck currently fails on existing Express 5 route-param typings across unrelated route files (for example server/src/http.ts and server/src/routes/*); none of those errors are in this PR's touched files. I did not include that unrelated cleanup here.

@bokelley

Copy link
Copy Markdown
Contributor Author

Updates noted. For reviewers: the npm run typecheck failures @bokelley flagged are a pre-existing repo-wide Express 5 route-param issue in files not touched by this PR — not a regression introduced here.


Generated by Claude Code

@aao-release-bot aao-release-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Follow-ups noted below. Three preconditions land where the spec puts them: capability + target-kind via TERMS_REJECTED, malformed payload via INVALID_REQUEST. Seller-level rollup mirrors the supported_optimization_metrics precedent.

Things I checked

  • Validation ordering in server/src/training-agent/task-handlers.ts:2082-2205. INVALID_REQUEST for shape misses (missing vendor.domain, missing metric_id, missing target.kind when target present, non-positive target.value); TERMS_REJECTED for capability and reporting-coherence misses. Line drawn at the same place as error-code.json description.
  • vendorMetricKey() collision behavior (absent brand_id vs "") at task-handlers.ts:181 — intentional, matches scripts/lint-vendor-metric-uniqueness.cjs:70 and the test at tests/lint-vendor-metric-uniqueness.test.cjs:31. Not a bug.
  • Schema-vs-docs coherence: seller-level supported_targets enum [cost_per, threshold_rate] in get-adcp-capabilities-response.json:578-581 matches the product-level enum at core/vendor-metric-optimization.json:29 and the worked example at docs/media-buy/media-buys/optimization-reporting.mdx:102. maximize_value correctly omitted (event-goal-only).
  • additionalProperties: true on the new object — matches the convention used by audience_management, conversion_tracking, frequency_capping. Forward-compat for a likely seller-level supported_metrics[] rollup later.
  • minor changeset classification. Repo is at 3.1.0-beta.3; additive optional field plus a validation path that only triggers on kind: "vendor_metric" (3.1-new) cannot break a buyer that wasn't already using the new goal kind. Correct.
  • Storyboard requires_capability: media_buy.vendor_metric_optimization (YAML L12-14) — gates against the seller-level rollup this PR just added. Right gate.
  • Test isolation. findProductWithVendorMetric() reads from buildCatalog(); seedVendorMetricProduct() uses unique product_ids per call; each test gets a fresh server = createTrainingAgentServer(DEFAULT_CTX). No state leakage across it.each cases.

Follow-ups (non-blocking — file as issues)

  • Docs lag the schema. docs/reference/whats-new-in-3-1.mdx:139 and docs/media-buy/media-buys/optimization-reporting.mdx:64 still describe only the per-product vendor_metric_optimization. The seller-level rollup added at get-adcp-capabilities-response.json:569-588 is on the wire but not in the prose. Mirror the supported_optimization_metrics row in the optimization-reporting table.
  • Fourth check is stricter than the docs normate. task-handlers.ts:2191-2202 rejects when a committed_metric is not in reporting_capabilities.vendor_metrics. Spec at optimization-reporting.mdx:79-81 documents only goal↔capability and goal↔committed_metrics. The check is defensible (the seller can't commit to a metric it doesn't list as reportable), but the storyboard's reject_unreportable_vendor_metric phase bakes it in as MUST. Either document this as a fourth precondition or widen the storyboard to allow seller acceptance.
  • TERMS_REJECTED enum description still scoped to measurement_terms. static/schemas/source/enums/error-code.json:138 still reads measurement_terms-only. After this PR, the code returns it for vendor-metric capability and coherence misses too. Widen the description in a follow-up so downstream sellers reading only the enum aren't surprised.
  • No guaranteed-delivery test for vendor_metric_optimization. product-factory.ts:565-568 explicitly does not gate vendor_metric_optimization on delivery_type — that's the design claim. Every seeded fixture in the test file uses delivery_type: 'non_guaranteed'. One guaranteed-delivery case would lock the claim.

Minor nits (non-blocking)

  1. Empty-string targetKind slips the supported_targets check. task-handlers.ts:2163 uses if (targetKind). Schema validation upstream rejects kind: "", but the defensive layer has a gap. Tighten the L2141 guard to also reject empty strings, or use if (typeof targetKind === 'string' && targetKind.length > 0) at L2163.
  2. target.value === 0 is rejected but untested. task-handlers.ts:2153 uses targetValue <= 0. The it.each at training-agent.test.ts:2550 only covers -1. Add a value: 0 case to close the boundary.
  3. inferVendorMetricBriefTerms regex asymmetry. task-handlers.ts:1037 matches /(attention|focus)/ but only adds the term attention. Works in practice (the metric_id token gets matched separately), but the asymmetry is a foot-gun if the identity tokens change later.
  4. brief_relevance grammar. Three-way join at task-handlers.ts:1554 produces Matches X channel(s) and keywords and vendor-metric optimization. — pre-existing pattern, two ands is ugly. Pre-existing.

Safe to merge.

@bokelley
bokelley merged commit 952787c into main May 24, 2026
26 checks passed
@bokelley
bokelley deleted the bokelley/vendor-metric-coverage branch May 24, 2026 11:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

compliance: add vendor_metric optimization-goal storyboard coverage

1 participant