Skip to content

fix(compliance): backport AgenticAdvertising.org suite fixture fixes#5231

Merged
bokelley merged 1 commit into
3.0.xfrom
backport/aao-compliance-suite-3.0
May 31, 2026
Merged

fix(compliance): backport AgenticAdvertising.org suite fixture fixes#5231
bokelley merged 1 commit into
3.0.xfrom
backport/aao-compliance-suite-3.0

Conversation

@bokelley

@bokelley bokelley commented May 31, 2026

Copy link
Copy Markdown
Contributor

Summary

  • backports the AgenticAdvertising.org compliance suite fixture fixes to 3.0.x source
  • moves media_buy_state_machine and measurement_terms_rejected setup windows to forward-looking Q3 2026 for the next 3.0.x release bundle
  • keeps existing 3.0.14 dist artifacts immutable; the fixed hosted 3.0 bundle should be produced by the next patch release
  • hardens storyboard idempotency freshness lint for mutating steps, including stable literal keys and duplicate generated aliases
  • makes newer optional compliance lints branch-tolerant when their scripts are absent on 3.0.x

Verification

  • npm run build:compliance
  • npm run test:storyboard-contradictions
  • media_buy_state_machine storyboard locally, legacy dispatch: 9 passed / 0 failed
  • media_buy_state_machine storyboard locally, framework dispatch: 9 passed / 0 failed
  • git diff --check
  • precommit hook: npm run test:unit, npm run test:test-dynamic-imports, npm run typecheck
  • pre-push hook: version sync

Release

  • Cut a 3.0.x patch release after merge so hosted 3.0 serves the fixed successor to 3.0.14.

@aao-release-bot aao-release-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes. The statusmedia_buy_status rename in state-machine.yaml is a 3.0.x wire-conformance regression — the matching schema additive-deprecate from #4904 didn't come back with it — and dist/protocol/3.0.14.tgz is now stale against the in-repo dist/compliance/3.0.14/ it bundles.

Blockers

  1. field_present: media_buy_status on a 3.0.x schema that only defines status. static/compliance/source/protocols/media-buy/state-machine.yaml:225,261,298 switch three update_media_buy validations from path: "status" to path: "media_buy_status". On 3.0.x, static/schemas/source/media-buy/update-media-buy-response.json:18 declares status (→ media-buy-status.json) and nothing else — zero media_buy_status hits anywhere under static/schemas/source/media-buy/. The companion additive-deprecate from #4904 that introduced media_buy_status alongside the deprecated status landed on main but was not backported. A 3.0.x seller emitting the schema-documented {media_buy_id, status: "paused"} fails all three checks. Two fixes: revert the three path: strings to "status" and ship a clean patch, or also backport #4904's schema rename and bump the changeset from patch to minor. A patch shipping a silent contract change on the conformance bar is the wrong shape. ad-tech-protocol-expert: unsound. code-reviewer: Critical.

  2. dist/protocol/3.0.14.tgz desynced from in-repo dist/compliance/3.0.14/**. scripts/build-protocol-tarball.cjs:185-189 bundles dist/compliance/{version}/ into the signed tarball. The PR rewrites the Q2→Q3 windows in dist/compliance/3.0.14/{protocols,domains}/media-buy/{state-machine,scenarios/measurement_terms_rejected}.yaml but leaves dist/protocol/3.0.14.tgz (sha256 80e2aca0…), .sha256, and .sig untouched — and the on-disk sha256 still matches the recorded one, so the tarball is internally consistent with its own old contents. SDK consumers that fetch by tarball get the expired May 2026 windows; consumers reading the unpacked tree get Q3. Re-tar and re-sign 3.0.14, or revert the dist/3.0.14 mutations and let 3.0.15 carry the window fix forward only.

Things I checked

  • runLintIfAvailable correctly no-ops the six lints not yet backported to 3.0.x (lint-compliance-packaged-refs.cjs, lint-vendor-metric-uniqueness.cjs, lint-storyboard-{check-enum,raw-mode-required,upstream-traffic-paths,advisory-expiry}.cjs). Branch-tolerance is the right shape for a backport.
  • New stableKeyViolations and duplicateGeneratedKeyViolations paths in lintStoryboardIdempotency are sound. universal/idempotency.yaml's deliberate alias-reuse allowance is preserved.
  • omit_idempotency_key === true replacing the prior blanket expect_error === true skip is the right tightening — it scopes the suppression to the explicit negative vector instead of every error path.
  • The Q2 → Q3 2026 window shift is load-bearing. Today is 2026-05-31; May 1–31 windows are expired.
  • required_any_of_tools frontmatter parsing matches the required_tools shape and forwards through generateIndex correctly.
  • is_fixed_price: true filter is in core/product-filters.json; pricing_options[0].fixed_price is the discriminator presence-check per core/pricing-option.json.
  • c30 → c28 is wire-legal — core/measurement-window.json defines window_id as freeform with examples only.

Follow-ups (non-blocking — file as issues)

  • runLintIfAvailable typo silent-skip. A misspelled scriptName produces only a (skipping …) log line forever. Gate by a hardcoded allowlist of known-missing scripts so a future rename on main doesn't disable a real lint on 3.0.x silently.
  • Duplicate-generated-key framing. The new branch is sound, but the prior aliases (measurement_terms_rejected_{aggressive,relaxed}_terms) were already unique on 3.0.x — no existing collision the fully-qualified rename resolves. Drop the collision framing from the changeset or cite the actual case.
  • published_version backfill. Newly-built indexes carry published_version; the shipped dist/compliance/3.0.14/index.json still doesn't. Either backfill or document the ≤3.0.14 fallback to adcp_version.

Minor nits (non-blocking)

  1. c30 → c28 in measurement_terms_rejected.yaml:125. Wire-legal, but unmentioned in the PR description and the changeset. One-line in-fixture comment would save the next reader the lookup.
  2. pricing_options[0].fixed_price check at measurement_terms_rejected.yaml:78. Brittle to sellers that don't sort the fixed option to the head of pricing_options. Pair with is_fixed_price: true filter assumption explicitly, or assert membership rather than index 0.

The PR description advertises three things and quietly does a fourth. Worth flagging so the next cherry-pick from main doesn't repeat the schema/storyboard split.

Request changes — restore 3.0.x storyboard/schema parity, then resync dist/protocol/3.0.14.tgz.

@bokelley bokelley force-pushed the backport/aao-compliance-suite-3.0 branch 2 times, most recently from f025471 to e80ebcc Compare May 31, 2026 22:04
@bokelley bokelley changed the title fix(compliance): backport AAO suite fixture fixes fix(compliance): backport AgenticAdvertising.org suite fixture fixes May 31, 2026

@aao-release-bot aao-release-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Date forward-port plus a tighter idempotency-key freshness lint, all on a branch where the optional sibling lints don't exist yet. Right shape for a 3.0.x backport.

Things I checked

  • The dates were genuinely stale. Today is 2026-05-31; old windows 2026-05-01..2026-05-31 were no longer forward-looking. Q3 2026 (2026-07-01..2026-09-30) is the correct shift across static/compliance/source/protocols/media-buy/scenarios/measurement_terms_rejected.yaml, static/compliance/source/protocols/media-buy/state-machine.yaml, and the four dist/compliance/3.0.14/** mirror files.
  • The expect_error === true blanket skip removal won't break the build. code-reviewer swept every expect_error: true step on 3.0.x: the only mutating one is the missing-key vector in universal/idempotency.yaml:176, which already carries omit_idempotency_key: true — the new narrower skip flag in scripts/build-compliance.cjs. Net behavior change: stable-key violations now get caught on expect_error paths too. Right direction.
  • runLintIfAvailable fail-closed semantics. scripts/build-compliance.cjs:42-61 exits 1 when an available lint script fails for all six wired callers; only the lint-storyboard-advisory-expiry.cjs invocation passes crashMessage to treat script crashes (not rule violations) as warnings. All six forward-ported scripts are absent on 3.0.x today, so the calls are no-ops until the scripts land — backport-safe.
  • Idempotency_key freshness lint is grounded in spec. ad-tech-protocol-expert confirmed the "$generate:uuid_v4#..." rule is the same MUST from docs/building/implementation/security.mdx:380 ("use a fresh UUID v4 for each request"), and the universal/idempotency.yaml carve-out at scripts/build-compliance.cjs:325 lines up with the existing replay-alias convention there.
  • reviewer_checks MUST is real, not invented. The new TERMS_REJECTED idempotency-claim-release MUST at static/compliance/source/protocols/media-buy/scenarios/measurement_terms_rejected.yaml:202 operationalizes security.mdx#idempotency rule 3 ("Only successful responses are cached") rather than inventing a stricter requirement.
  • measurement_window: "c28" is schema-valid. core/measurement-terms.json:22-35 treats measurement_window as a free-form window_id string; c28 is no less valid than c30. The new field_present check on products[0].pricing_options[0].fixed_price matches core/product-filters.json:15 semantics.
  • Changeset present and correct type. .changeset/aao-compliance-suite-3-0-fixes.md, patch for adcontextprotocol. Right bump for a fixture/lint fix.
  • No schema source changes. static/schemas/source/** is untouched; this is fixtures + build-script polish, no wire shape moves.

Follow-ups (non-blocking — file as issues)

  • required_any_of_tools has zero downstream consumers today. grep -rn required_any_of_tools finds only the new build-script lines — no specialism declares it, no SDK reads it, no schema describes it. Forward-compatible (emitted only when present), but the 3.0.x backport is an odd first carrier for a wire field with no reader. Open a @adcp/sdk tracking issue or pull the field forward only.
  • published_version + adcp_version is a dual-write with no deprecation timer. Both carry full semver per scripts/build-compliance.cjs:504. Fine today, but the comment says "legacy alias kept through 3.x for @adcp/client compatibility" without a 4.x removal note in the spec or a @deprecated marker on ComplianceIndex. Either annotate the JSON schema or flag the 4.0 migration guide.
  • runLintIfAvailable path coupling. The existence check uses __dirname/<scriptName> but the exec uses cd __dirname/.. && node scripts/<scriptName>. The two will diverge silently if anyone moves build-compliance.cjs. Invoke node <scriptPath> directly to keep them in lockstep.

Approving on the strength of the spec-grounded MUSTs (security.mdx:380, rule 3) plus the verified lint-skip coverage of all 16 expect_error steps on 3.0.x. Cut the 3.0.x patch after merge so hosted /compliance/3.0/ serves the corrected windows.

@aao-release-bot aao-release-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Clean backport with two genuine hardenings on top (stable-key + duplicate-alias detection). Follow-ups noted below.

Things I checked

  • All expect_error: true steps on 3.0.x are safe under the new lint contract — every one either declares idempotency_key, points at a non-mutating schema, uses a placeholder task not in mutatingTools, or sets omit_idempotency_key: true (only universal/idempotency.yaml:176). The behavior change at scripts/build-compliance.cjs:298-313 doesn't break any existing storyboard.
  • No stable (non-$generate:) idempotency keys anywhere in static/compliance/source/**. The new stableKeyViolations array is empty on this branch.
  • The only duplicate $generate:uuid_v4#alias use is inside universal/idempotency.yaml (the replay storyboard), which the new lint hard-allowlists.
  • is_fixed_price exists on static/schemas/source/core/product-filters.json on 3.0.x; fixed_price exists on the pricing-option schemas. The new fixture validates.
  • c28 is valid: measurement_window is unconstrained string with non-normative examples.
  • All six conditionally-invoked lint scripts (lint-compliance-packaged-refs.cjs and friends) are absent on 3.0.x, so runLintIfAvailable correctly skips them — this is the intent per the PR description.
  • state-machine.yaml field_present: status assertions are already correct against update-media-buy-response.json oneOf[0].properties.status — no assertion change was needed, only the dates.

Follow-ups (non-blocking — file as issues)

  • runLintIfAvailable silently drops 6 protections on 3.0.x patch releases. scripts/build-compliance.cjs:42-62. Each skip is one stdout line; in CI logs that's easy to miss. Worth either a single aggregate WARN summarizing all skipped lints, or a line in the changeset that 3.0.x patches ship without packaged-ref / vendor-metric-uniqueness / check-enum / raw-mode-required / upstream-traffic-paths / advisory-expiry coverage.
  • Changeset overstates the state-machine change. .changeset/aao-compliance-suite-3-0-fixes.md says "state-machine assertions match the 3.0.x status response field" but the diff is purely date moves (2026-052026-07). Reviewers reading the changelog will look for assertion changes and find none. Rephrase to "state-machine fixture dates move to a forward-looking flight window; existing status assertions remain correct on 3.0.x."
  • Narrative contradicts reviewer_check in measurement_terms_rejected.yaml. Narrative says "reusing the prior key against a different body MUST be rejected with IDEMPOTENCY_CONFLICT per spec." Per security.mdx#idempotency rule 3 ("Only successful responses are cached"), a TERMS_REJECTED first call never populates the cache — so the same-key retry MUST succeed, which is exactly what the new reviewer_checks block on create_media_buy_relaxed_terms (lines 201-204) probes. Either drop the IDEMPOTENCY_CONFLICT sentence from the narrative or qualify it to "after a successful first call." On a backport whose whole point is getting this rule right, the narrative shouldn't teach the wrong invariant.
  • Duplicate-alias escape hatch is hard-coded path equality. scripts/build-compliance.cjs:333 uses rel !== 'universal/idempotency.yaml'. A future per-task replay storyboard outside that file has no escape hatch other than amending the lint. Per-step share_idempotency_alias: true or a top-level allowlist constant would age better. Non-urgent — universal/idempotency.yaml is the only replay storyboard today.

Minor nits (non-blocking)

  1. Stale @adcp/sdk@5.x rename. scripts/build-compliance.cjs (the renamed transitional-alias comment): the repo uses @adcp/client everywhere (43+ imports across server/, scripts/, docs/), and the newly-added published_version comment block correctly references @adcp/client. One comment line was renamed to @adcp/sdk@5.x, leaving the file internally inconsistent. On a 3.0.x backport where the published SDK is still @adcp/client, revert that one line.
  2. Positional assumption in fixed-price check. measurement_terms_rejected.yaml:78-80: field_present at products[0].pricing_options[0].fixed_price assumes the first pricing option is the fixed-price one. The is_fixed_price: true filter only guarantees at least one option qualifies — agents are free to return auction options first. A JSONPath query (pricing_options[?(@.fixed_price)]) or a description that flags the positional assumption is safer.
  3. Path-separator portability. scripts/build-compliance.cjs:333path.relative returns native separators; rel !== 'universal/idempotency.yaml' would mis-compare on Windows. CI is Linux, so this is a portability nit, not a defect.

ad-tech-protocol-expert: sound-with-caveats — no spec divergence; the only normative issue is the narrative contradiction noted above. code-reviewer: no blockers; the lint hardening is correctly anchored against the existing source tree.

Approving on the strength of the lint hardening plus the verified all-clear sweep across expect_error and $generate: use sites. Ship the patch release once CI confirms the build skips the six absent lints cleanly. The changeset and narrative wording are worth a one-commit polish before tagging — notable that a PR specifically backporting an idempotency rule ships with a narrative that contradicts it.

@bokelley bokelley force-pushed the backport/aao-compliance-suite-3.0 branch from e80ebcc to d1afe81 Compare May 31, 2026 22:24
@bokelley bokelley force-pushed the backport/aao-compliance-suite-3.0 branch from d1afe81 to d6db4d4 Compare May 31, 2026 22:25

@aao-release-bot aao-release-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Follow-ups noted below. Clean backport — the idempotency narrative rewrite is the load-bearing fix and matches the 3.0.x spec.

Things I checked

  • scripts/build-compliance.cjs:296 — old if (step.expect_error === true) continue; removed. New omit_idempotency_key === true opt-out at L316 is a stricter, intent-declaring surface. Walked every expect_error: true storyboard (15 files); the only one with a key in sample_request is universal/idempotency.yaml:457 ($generate:uuid_v4#replay_key), exempted by the rel !== 'universal/idempotency.yaml' carve-out at the duplicate-alias check. Behavior is correct.
  • scripts/build-compliance.cjs:42-62runLintIfAvailable swallows non-zero exits only when crashMessage is set, otherwise process.exit(1). Only lint-storyboard-advisory-expiry.cjs passes a crashMessage, matching the comment at L678-685 ("script exits 0 on rule violations; non-zero exit means script crashed").
  • lintStoryboardIdempotency rewrite — isGeneratedIdempotencyKey at L221 matches both bare $generate:uuid_v4 and #alias form. No anchoring bug.
  • readYamlFrontmatter required_any_of_tools validation — array, non-empty, each entry {tools: string[≥2], rationale?: string}. Error messages cite path + index. Sound.
  • YAML date shifts — all three fixtures move Q2 (May 2026) → Q3 (Jul-Sep 2026). start < end, same calendar year, coherent.
  • measurement_terms_rejected.yaml narrative — old text claimed key reuse against a different body MUST be rejected with IDEMPOTENCY_CONFLICT after TERMS_REJECTED. That contradicts security.mdx rule 3 (only successful responses are cached) — a rejected response leaves no canonical hash to conflict against. New text is the correct contract. The added reviewer_checks clause requiring a fresh media_buy_id on the second attempt is the right enforcement signal.
  • c30c28static/schemas/source/core/measurement-terms.json:22 defines measurement_window as free-form string; c3/c7 are examples, not enum members. c28 aligns with the 28-day digital reconciliation window. Fixture hygiene win.
  • Changeset present, descriptively named, patch is defensible (additive index.json fields, no wire-schema change).

Follow-ups (non-blocking — file as issues)

  • published_version and required_any_of_tools are 3.1.0-beta surfaces. Both land in 3.0.x bundles via this PR without a 3.0.x storyboard-schema.yaml definition or ComplianceIndex schema update on 3.0.x. The parser is dormant until a 3.0.x specialism opts in, but adopters that key off index.json see new fields without a doc anchor. Consider landing the schema/doc update on 3.0.x in a follow-up patch, or gating the emit on a version check.
  • Six branch-tolerant lint skips warrant WG sign-off. lint-compliance-packaged-refs, lint-vendor-metric-uniqueness, lint-storyboard-check-enum (#3830), lint-storyboard-raw-mode-required (#3847 item 2), lint-storyboard-upstream-traffic-paths (#5073), lint-storyboard-advisory-expiry — all silently skip on 3.0.x. That means 3.0.x publishing tolerates check-enum typos, raw-mode misuse, and identifier_path grammar drift that the WG already deemed errors on main. Acceptable for a backport patch; flag the policy decision explicitly.

Minor nits (non-blocking)

  1. runLintIfAvailable parameter shape. crashMessage is implicitly a boolean policy switch (truthy → swallow, falsy → exit 1). { allowCrash: true, message: '...' } or a crashAdvisory rename would make the policy harder to misuse for future call sites. scripts/build-compliance.cjs:42.
  2. required_any_of_tools.tools has no internal de-dup. A future author writing tools: ["x", "x", "y"] passes the ≥2 gate while semantically being a single-tool family. scripts/build-compliance.cjs:121.

Approving on the strength of the idempotency-narrative correction and the lint tightening. Cut the 3.0.x patch after merge so hosted 3.0 serves the fixed successor to 3.0.14.

@aao-release-bot aao-release-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. The backport is clean and the narrative correction is load-bearing — the old fixture was teaching a spec error.

Things I checked

  • omit_idempotency_key: true is set at static/compliance/source/universal/idempotency.yaml:176, so swapping the lint opt-out from expect_error to omit_idempotency_key doesn't regress the missing-key negative vector. The conflict step keeps a valid idempotency_key and wouldn't trip the lint either.
  • New narrative on measurement_terms_rejected.yaml matches docs/building/implementation/security.mdx#idempotency rule 3 ("Only successful responses are cached"). The old narrative was wrong — TERMS_REJECTED is an error, the key claim is not retained, the buyer can legitimately retry the same key with corrected terms. The reviewer_checks cite is well-anchored.
  • measurement_window is free-form string in core/measurement-terms.json / core/measurement-window.json, not a closed enum — c30c28 is a fixture-content choice, not spec drift.
  • is_fixed_price is a documented boolean filter in core/product-filters.json:15; pricing_options[].fixed_price is documented across the pricing-options schemas.
  • duplicateGeneratedKeyViolations correctly exempts universal/idempotency.yaml at the populate-and-check site in scripts/build-compliance.cjs. Cross-file alias collisions outside that file get caught on the second occurrence.
  • Changeset is patch and the wire surface is unchanged — fixtures + build script + additive published_version field on index.json. Right bump.

Follow-ups (non-blocking — file as issues)

  • runLintIfAvailable crashMessage branch in scripts/build-compliance.cjs prints to stderr and returns. The caller comment says script crashes "SHOULD surface — don't silently swallow implementation bugs that mask the lint forever." With the execSync error consumed and main() continuing, CI exits 0 — the crash is exactly as silent as the comment warns against. On 3.0.x this is academic (the script doesn't exist here). On main it isn't. Change to process.exitCode = 1 so the build keeps running but the run goes red.
  • published_version is additive and back-compat for @adcp/client consumers reading adcp_version, but worth one line in the changeset documenting the new key so the next consumer doesn't discover it by accident.
  • reviewer_checks is established by precedent (universal/idempotency.yaml:486 plus this fixture) but isn't declared in storyboard-schema.yaml. Worth declaring formally as a step-level array of strings.

Minor nits (non-blocking)

  1. Narrative says "same key"; fixture uses two aliases. The measurement_terms_rejected.yaml narrative + reviewer_checks describe a same-key retry across reject_termsaccept_terms, but the executable steps use distinct $generate:uuid_v4#...aggressive_terms / ...relaxed_terms aliases — correct, because the new duplicate-alias lint would block same-aliases outside universal/idempotency.yaml. One inline comment on the fixture pointing at the deliberate split would stop a future maintainer from "fixing" the alias mismatch and tripping the lint.

LGTM. Follow-ups noted below.

@bokelley bokelley merged commit f78f37c into 3.0.x May 31, 2026
13 checks passed
@bokelley bokelley deleted the backport/aao-compliance-suite-3.0 branch May 31, 2026 22:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant